[Test][gtest] Sl/test solver #1877

xinlipn · 2022-11-14T05:20:15Z

ConvAsm3x3 solver test. Passed CI after disabling Navi21 which is not supported by this solver.

atamazov · 2022-11-15T15:12:53Z

test/test.hpp

+// Conflicts with GoogleTest FAIL()
+#ifdef FAIL
+#undef FAIL


How it could be so that test/test.hpp is used in gtest?

@atamazov test.hpp isn't used in GTest, however, there's a warning from compiler about FAIL() being already defined in GoogleTest. This change probably shouldn't go to develop

Perhaps we can refactor these defs out into their own header, so that they are only included when they are needed

Perhaps we can refactor these defs out into their own header, so that they are only included when they are needed

@JehandadKhan separated Macros defined in test.hpp to a separate header test_macro.hpp and updated affected code.

atamazov · 2022-11-15T15:19:13Z

This can't be merged in due to Navi21 issues. I recommend converting this into draft.

atamazov · 2022-11-15T15:20:38Z

@junliume @JehandadKhan I highly recommend pausing with new GoogleTests until #1843 (comment) is resolved.

atamazov · 2022-11-15T15:21:48Z

@xinlipn

ConvAsm3x3 solver test.

Please describe which aspects of the solver's behavior this test is intended to check. Thanks.

by @xinlipn:
@atamazov the intension is to bring test down to solver level for more granularity, i.e. each solver test will have its own gtest. Hope this makes sense.

xinlipn · 2022-11-15T17:37:29Z

This can't be merged in due to Navi21 issues. I recommend converting this into draft.

@atamazov ConvAsm3x3 doesn't support Navi21, only supports gfx8* and gfx9*.

atamazov · 2022-11-16T17:23:44Z

@xinlipn That is why this test won't pass on Navi21. If this PR is merged in, CI will always fail. Which means that this PR, in its current state, shouldn't be merged.

atamazov · 2022-11-16T17:25:55Z

@xinlipn

@atamazov the intension is to bring test down to solver level for more granularity, i.e. each solver test will have its own gtest. Hope this makes sense.

Yes, this makes sense. However, this does not answer my question.

pfultz2 · 2022-11-16T18:04:05Z

If we are writing tests for solvers then there should be unit tests for IsApplicable, GetSolution, and IsValidPerformanceConfig to get full branch coverage. For example, there is a lot of checking that is in IsApplicable, I would expect a unit test for each one of these branches. These unit tests should be very fast so it should easy to run through many configs to check all the branches.

JehandadKhan · 2022-11-16T22:25:48Z

test/gtest/solver_convasm3x3u.cpp

+    if(!solv.IsApplicable(ctx))
+    {
+        test_skipped = true;
+        GTEST_FAIL() << solv.SolverDbId() << "ConvAsm3x3U Not Applicable for this problem"


Suggested change

GTEST_FAIL() << solv.SolverDbId() << "ConvAsm3x3U Not Applicable for this problem"

GTEST_SKIP() << solv.SolverDbId() << "ConvAsm3x3U Not Applicable for this problem"

@JehandadKhan replaced with GTEST_SKIP()

Good. This change resolves #1877 (comment).

JehandadKhan · 2022-11-16T23:13:04Z

This can't be merged in due to Navi21 issues. I recommend converting this into draft.

That is a false issue, if the solver is not applicable on Navi, then the test should skip instead of failing, since there is nothing to test! Please see my suggestion here

@junliume @JehandadKhan I highly recommend pausing with new GoogleTests until #1843 (comment) is resolved.

I dont agree with that, since that would mean a new test which would enhance test coverage in MIOpen would be withering in our PR section. Since we wish to get to a place where we test each solver individually.

If we are writing tests for solvers then there should be unit tests for IsApplicable, GetSolution, and IsValidPerformanceConfig to get full branch coverage. For example, there is a lot of checking that is in IsApplicable, I would expect a unit test for each one of these branches. These unit tests should be very fast so it should easy to run through many configs to check all the branches.

I agree, however, that is not the intent of this PR. The purpose of this PR to increase numerical accuracy coverage to a new solver, which previously does not exists. The current testing infra uses the API to test numerical accuracy on the API level where individual solvers might get skipped since they were not fast enough. Once we have numerical coverage over all the solvers, we can start the work on unit testing each solver as you propose.

cc @xinlipn

…oid conflicts in GTest

atamazov · 2022-11-17T12:36:48Z

test/test_macro.hpp

+
+#include "test.hpp"
+
+#define CHECK(...)                                    \


Why this is copied from test.hpp?

Why this is copied from test.hpp?

@atamazov this idea is to separate FAIL() out in a separate header which conflicts with the same definition in GoogleTest. It seems to be strange if just move FAIL() out, that's I put all Macros in the new header file

@xinlipn But then why test.hpp (that contains the same CHECK) is included here?

@xinlipn But then why test.hpp (that contains the same CHECK) is included here?

@atamazov separate out only one Macro FAIL() from the rest and just create a header file for it seems to be a temp workaround and looks weird logically, at least to me.

FAIL() isn't used anywhere and I removed it for now to minimize the impact to other code.
Consequently test_macro.hpp is removed and changes in other headers anyway are reverted.

atamazov · 2022-11-17T12:36:53Z

test/test_macro.hpp

+                                                      \
+    } while(false)
+
+#define EXPECT(...)                                         \


Please see comments above

atamazov · 2022-11-17T12:36:59Z

test/test_macro.hpp

+                                                            \
+    } while(false)
+
+#define EXPECT_OP(LEFT, OP, RIGHT)                                 \


Please see comments above

atamazov · 2022-11-17T12:38:09Z

test/handle_test.cpp

@@ -39,7 +39,7 @@
 #include "get_handle.hpp"
 #include <vector>
 #include <thread>
-#include "test.hpp"
+#include "test_macro.hpp"


Any way to resolve the "FAIL" problem without touching existing tests?

@atamazov another way is only separate FAIL in to a new header file or remove this definition for now if no code is currently using it.

The PR can be simplified, if we just remove the FAIL macro if its not being used anywhere

The PR can be simplified, if we just remove the FAIL macro if its not being used anywhere

@JehandadKhan done
cc @atamazov

JehandadKhan · 2022-11-17T19:12:49Z

test/gtest/solver.hpp

+        miopenDestroyConvolutionDescriptor(conv_desc);
+    }
+    ConvTestCase conv_config;
+    miopenConvolutionDescriptor_t conv_desc;


If we use the MIOpen C++ objects instead, then you would not need to manage the lifetime of these objects.

If we use the MIOpen C++ objects instead, then you would not need to manage the lifetime of these objects.

@JehandadKhan , done

…rs, Init desc using Object

atamazov

No objections!

xinlipn added 5 commits October 30, 2022 22:34

Add ConASM3x3U solver test, need to update setup params

6301c5f

added miopenExecuteFusionPlan

5723f77

Updated ConvASM3x3 solver test, rm FAIL conflict with GoogleTest

4df90ab

Remove warning

c60ffb8

Split common code to header, updated test code

87dafc0

xinlipn requested a review from JehandadKhan November 14, 2022 05:21

xinlipn force-pushed the sl/test_solver branch from 8700896 to 87dafc0 Compare November 15, 2022 01:21

Add more test cases and clean up

2a60e9c

atamazov suggested changes Nov 15, 2022

View reviewed changes

JehandadKhan reviewed Nov 16, 2022

View reviewed changes

Replace GTEST_FAIL with GTEST_SKIP, separate Macros in test.hpp to av…

7c757be

…oid conflicts in GTest

atamazov reviewed Nov 17, 2022

View reviewed changes

Update code to skip Navi21 that's not supported, clean code

3afa516

JehandadKhan reviewed Nov 17, 2022

View reviewed changes

Remove FAIL to avoid conflicts in GoogleTest, Revert changes in heade…

6ed947d

…rs, Init desc using Object

JehandadKhan approved these changes Nov 18, 2022

View reviewed changes

atamazov approved these changes Nov 21, 2022

View reviewed changes

junliume approved these changes Nov 22, 2022

View reviewed changes

junliume changed the title ~~Sl/test solver~~ [Test][gtest] Sl/test solver Nov 22, 2022

junliume approved these changes Nov 22, 2022

View reviewed changes

junliume merged commit cbc1665 into develop Nov 22, 2022

junliume deleted the sl/test_solver branch March 23, 2023 20:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Test][gtest] Sl/test solver #1877

[Test][gtest] Sl/test solver #1877

xinlipn commented Nov 14, 2022

atamazov Nov 15, 2022

xinlipn Nov 15, 2022 •

edited

Loading

JehandadKhan Nov 16, 2022

xinlipn Nov 17, 2022

atamazov commented Nov 15, 2022

atamazov commented Nov 15, 2022

atamazov commented Nov 15, 2022 •

edited

Loading

xinlipn commented Nov 15, 2022

atamazov commented Nov 16, 2022 •

edited

Loading

atamazov commented Nov 16, 2022

pfultz2 commented Nov 16, 2022

JehandadKhan Nov 16, 2022

xinlipn Nov 17, 2022

atamazov Nov 17, 2022

JehandadKhan commented Nov 16, 2022

atamazov Nov 17, 2022

xinlipn Nov 17, 2022

atamazov Nov 17, 2022

xinlipn Nov 18, 2022

atamazov Nov 17, 2022

xinlipn Nov 17, 2022

atamazov Nov 17, 2022

xinlipn Nov 17, 2022

atamazov Nov 17, 2022

xinlipn Nov 17, 2022

JehandadKhan Nov 17, 2022

xinlipn Nov 18, 2022

JehandadKhan Nov 17, 2022

xinlipn Nov 18, 2022

atamazov left a comment

	GTEST_FAIL() << solv.SolverDbId() << "ConvAsm3x3U Not Applicable for this problem"
	GTEST_SKIP() << solv.SolverDbId() << "ConvAsm3x3U Not Applicable for this problem"

[Test][gtest] Sl/test solver #1877

[Test][gtest] Sl/test solver #1877

Conversation

xinlipn commented Nov 14, 2022

Choose a reason for hiding this comment

xinlipn Nov 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atamazov commented Nov 15, 2022

atamazov commented Nov 15, 2022

atamazov commented Nov 15, 2022 • edited Loading

xinlipn commented Nov 15, 2022

atamazov commented Nov 16, 2022 • edited Loading

atamazov commented Nov 16, 2022

pfultz2 commented Nov 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JehandadKhan commented Nov 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atamazov left a comment

Choose a reason for hiding this comment

xinlipn Nov 15, 2022 •

edited

Loading

atamazov commented Nov 15, 2022 •

edited

Loading

atamazov commented Nov 16, 2022 •

edited

Loading