[NPU] Add CLI option to directly pass reference files when running tests #28446

seowwj · 2025-01-15T06:53:44Z

Details:

Enables user to directly specify reference files to be used in test mode. If none is specified, the current default file scheme (<netname>_ref_out....blob) is used.

Tickets:

E#148130

Maxim-Doronin · 2025-01-15T11:16:10Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

@@ -127,6 +127,7 @@ DEFINE_string(
    ref_dir,
    "",
    "A directory with reference blobs to compare with in run_test mode. Leave it empty to use the current folder.");
+DEFINE_string(ref_results, "", "Reference files used during 'run_test'");


Please, specify input format in case of several inputs.
Is it --ref_results "file1;file2"?

Are relative paths supported? e.g. --ref_dir /part/of/path/ --ref_results rest/of/path/file.blob?

Are absolute paths supported? e.g. --ref_results /abs/path/file.blob

Maxim-Doronin · 2025-01-15T11:20:03Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

+            // Make sure that the number of reference files is the same as the number of input files
+            if (refFilesPerCase.size() != inputFilesPerCase.size()) {
+                std::cout << "The number of reference files is not equal to the number of input files."
+                    << "  # Reference Files: " << refFilesPerCase.size()


Please replace # with number of or # of

Maxim-Doronin · 2025-01-15T11:21:59Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

+            }
+
+            for (const auto& refResult : refFilesPerCase) {
+                std::vector<std::string> refFilesPerModel = splitStringList(refResult, ',');


AFAIR, --network supports only a single model to be loaded. Please explain when we expect several models in a single SIT run.

I guess that it is for batched input scenario when a whole row looks like CASE_0;CASE_1;etc
where
CASE_X is a run of SIT and which contains INPUT_0,INPUT_1,INPUT2,etc,
where
INPUT_0 encompasses data for single input in batch BATCH_0|BATCH_1|etc
which makes sense for separation input data per model

If my understanding is correct that here we need to produce tensors which are belong to output data types. In case of batch, though inputs pictures for every batch line are separated, outputs are printed into unified single file without separation of its lines. So probably we should not use this level with '|' as perFiles separator

I guess that it is for batched input scenario when a whole row looks like CASE_0;CASE_1;etc where CASE_X is a run of SIT and which contains INPUT_0,INPUT_1,INPUT2,etc, where INPUT_0 encompasses data for single input in batch BATCH_0|BATCH_1|etc which makes sense for separation input data per model

This is correct, the separators are mainly for delimiting if there are multiple expected outputs for the model.

I see, but in order to leverage that "multiple expected outputs for the model" you need to write a special logic to split "sigle batched output" onto several non-batched outputs - in order to make it works. Without that logic implemented this "|" is confusing as it doesn't work properly

sivanov-work · 2025-01-15T11:49:31Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

+            }
+
+            for (const auto& refResult : refFilesPerCase) {
+                std::vector<std::string> refFilesPerModel = splitStringList(refResult, ',');


I guess that it is for batched input scenario when a whole row looks like CASE_0;CASE_1;etc
where
CASE_X is a run of SIT and which contains INPUT_0,INPUT_1,INPUT2,etc,
where
INPUT_0 encompasses data for single input in batch BATCH_0|BATCH_1|etc
which makes sense for separation input data per model

If my understanding is correct that here we need to produce tensors which are belong to output data types. In case of batch, though inputs pictures for every batch line are separated, outputs are printed into unified single file without separation of its lines. So probably we should not use this level with '|' as perFiles separator

sivanov-work · 2025-01-15T11:56:41Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

+                    if (!refFiles.empty()) {
+                        blobFileFullPath = refFiles[numberOfTestCase][outputInd];
+                    } else {
+                        std::ostringstream ostr;


nitpick:
I'd rather move this into a separate function like 'generate_reference_file_name(...)', which will makes this huge code lesser overwhelming

nitpick 2:
I wondered whether forming the array refFiles initially regardless either ref_dir or res_output set and stick to it will without this condition checking refFiles.empty() would be more nice or not.
So that we could fill refFiles by using generate_reference_file_name() (because all data to generate is known in the beginning) when ref_dir exist or just parse command line of res_output

Please feel free to ignore this comment as it's arguable

sivanov-work · 2025-01-15T11:59:36Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

-                    const auto blobFileFullName = fullPath.string();
+                    // If reference files are provided, use them
+                    if (!refFiles.empty()) {
+                        blobFileFullPath = refFiles[numberOfTestCase][outputInd];


I found that we have already taken numberOfTestCase here https://github.com/openvinotoolkit/openvino/pull/28446/files#diff-32f65067f63ae981b516b3780122e243b0ed1684efd43992f00063f0d458716eR2118
when we had defined refFiles by itself. Probably when use remove '|' parsing level this second numberOfTestCase would not required anymore

sivanov-work · 2025-01-15T12:02:31Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

+                                                                                 : refFilesForOneInfer[numberOfTestCase];
+            if (!FLAGS_ref_results.empty()) {
+                OPENVINO_ASSERT(refFiles.size() == outputsInfo.size(), "Number of reference files ", refFiles.size(),
+                " doesn't match network configuration ", outputsInfo.size());


+100500 for printing troubleshooting info's!
I'd only suggest emphasize that it must match output count by amending this doesn't match network output count: ', outputsInfo.size()

sivanov-work · 2025-01-15T12:10:47Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

-        std::cout << "    Reference files direcotry:                "
-                  << (FLAGS_ref_dir.empty() ? "Current directory" : FLAGS_ref_dir) << std::endl;
+        std::cout << "    Reference files directory:                "
+                  << (FLAGS_ref_dir.empty() && FLAGS_ref_results.empty() ? "Current directory" : FLAGS_ref_dir)


from a context it sounds like these parameters are mutually exclusive so that I presume that printing a message and exiting with an error is more intentional

I would argue that they are not mutually exclusive in this case.
Case 1: User provides a reference directory + relative file paths
Result: This line should show the reference directory given

Case 2: User only provides a reference directory
Result: This line should show the reference directory given

Case 3: Nothing provided
Result: This line should show "Current directory"

💡Perhaps it would be even better to show the actual directory path instead of just "current directory"?

it's the question about UX: for me mostly as a UNIX user it's quite obvious that if the path starting from "/" it must be an absolute path, so that these parameters --ref_dir=A --ref_outputs=/B;/C mean that /B and /C are expected to be absolute paths rather then A/B and A/C. Thus, if we can configure either absolute or relative paths using only one parameter, instead of changing behavior of that parameter by looking at a value of another one parameter - it will be more natural in usage

this is IMHO, so that maybe it would be better to ask an issue reporter opinion

…tch network output

sivanov-work · 2025-01-21T11:24:35Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

+            }
+
+            for (const auto& refResult : refFilesPerCase) {
+                std::vector<std::string> refFilesPerModel = splitStringList(refResult, ',');


I see, but in order to leverage that "multiple expected outputs for the model" you need to write a special logic to split "sigle batched output" onto several non-batched outputs - in order to make it works. Without that logic implemented this "|" is confusing as it doesn't work properly

sivanov-work · 2025-01-21T11:31:47Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

@@ -2084,6 +2145,12 @@ static int runSingleImageTest() {
            const FilesForModelInputs &inputFiles = inputFilesForOneInfer[numberOfTestCase];
            OPENVINO_ASSERT(inputFiles.size() == inputsInfo.size(), "Number of input files ", inputFiles.size(),
                            " doesn't match network configuration ", inputsInfo.size());
+            const RefFilesForModelOutputs &refFiles = refFilesForOneInfer.empty() ? RefFilesForModelOutputs{}
+                                                                                 : refFilesForOneInfer[numberOfTestCase];


I still didn't get why we are getting this refFiles as refFilesForOneInfer[numberOfTestCase]; and then getting
refFiles[numberOfTestCase] again inside a function getRefBlobFilePath()

I assume that there is the confusion regarding this "|" layer in parsing refFilesForOneInfer, which brings up this addition array layer

I have removed the "|" layer as it may be less useful in model output.
Hence now the convention becomes: Ref1A, Ref1B; Ref2A, Ref2B

Ref1A, Ref1B will constitute the first test case (test case 0), with 2 outputs (1A and 1B)
Ref2A, Ref2B will constitute the second test case (test case 1), with 2 outputs (2A and 2B)

I've also cleared up the confusion in the help text. (semicolon = different test case, comma = different reference file within same test case)

sivanov-work

LGTM

src/plugins/intel_npu/tools/single-image-test/main.cpp

MikhailPedus

LGTM

Add CLI option to directly pass reference files when running tests

7280beb

seowwj requested review from a team as code owners January 15, 2025 06:53

github-actions bot added the category: NPU OpenVINO NPU plugin label Jan 15, 2025

sys-openvino-ci added the ExternalPR External contributor label Jan 15, 2025

Maxim-Doronin added good first issue Good for newcomers and removed ExternalPR External contributor labels Jan 15, 2025

Maxim-Doronin reviewed Jan 15, 2025

View reviewed changes

sivanov-work reviewed Jan 15, 2025

View reviewed changes

Improve reference files to accept both relative and absolute paths

2a00e10

sys-openvino-ci added the ExternalPR External contributor label Jan 16, 2025

Clarify error message to explain when number of ref files does not ma…

95404ba

…tch network output

seowwj changed the title ~~Add CLI option to directly pass reference files when running tests~~ [NPU] Add CLI option to directly pass reference files when running tests Jan 17, 2025

Move reference blob path logic to own function

1a4dacb

sivanov-work reviewed Jan 21, 2025

View reviewed changes

seowwj and others added 3 commits January 28, 2025 11:55

Merge branch 'openvinotoolkit:master' into npu-SIT-pass-ref-results

4b38006

Simplify reference files indexing

f2e98a9

Corrects mistake in ref_results help message

9d6e075

sivanov-work approved these changes Jan 28, 2025

View reviewed changes

src/plugins/intel_npu/tools/single-image-test/main.cpp Outdated Show resolved Hide resolved

Using platform agnostic method to construct file path

3938f80

DariaMityagina approved these changes Feb 10, 2025

View reviewed changes

Maxim-Doronin approved these changes Feb 10, 2025

View reviewed changes

Merge branch 'master' into npu-SIT-pass-ref-results

33f81c0

MikhailPedus reviewed Feb 10, 2025

View reviewed changes

Merge branch 'master' into npu-SIT-pass-ref-results

abb9035

Maxim-Doronin enabled auto-merge February 11, 2025 10:41

Maxim-Doronin added this pull request to the merge queue Feb 12, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Feb 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NPU] Add CLI option to directly pass reference files when running tests #28446

[NPU] Add CLI option to directly pass reference files when running tests #28446

seowwj commented Jan 15, 2025 •

edited by Maxim-Doronin

Loading

Maxim-Doronin Jan 15, 2025

Maxim-Doronin Jan 15, 2025

Maxim-Doronin Jan 15, 2025

sivanov-work Jan 15, 2025

seowwj Jan 21, 2025

sivanov-work Jan 21, 2025

sivanov-work Jan 15, 2025

sivanov-work Jan 15, 2025

sivanov-work Jan 15, 2025

sivanov-work Jan 15, 2025

sivanov-work Jan 15, 2025

sivanov-work Jan 15, 2025

seowwj Jan 21, 2025

seowwj Jan 21, 2025

sivanov-work Jan 21, 2025 •

edited

Loading

sivanov-work Jan 21, 2025

sivanov-work Jan 21, 2025

seowwj Jan 28, 2025

sivanov-work left a comment

MikhailPedus left a comment

[NPU] Add CLI option to directly pass reference files when running tests #28446

Are you sure you want to change the base?

[NPU] Add CLI option to directly pass reference files when running tests #28446

Conversation

seowwj commented Jan 15, 2025 • edited by Maxim-Doronin Loading

Details:

Tickets:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sivanov-work Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sivanov-work left a comment

Choose a reason for hiding this comment

MikhailPedus left a comment

Choose a reason for hiding this comment

seowwj commented Jan 15, 2025 •

edited by Maxim-Doronin

Loading

sivanov-work Jan 21, 2025 •

edited

Loading