Optimize the model analysis weekly pipeline

[chandrasekaranpradeep]

The PR will optimise the model analysis weekly pipeline - #761

The current model_analysis script will generate the unique op tests and running the unique op configuration test model wise instead collect the unique op test configuration from all the models and then extract unique op configuration test across all the models which will avoid running same configuration that are present in multiple models again and again.

For example, if we have add op with operands shape of (1, 32) and (1, 32) in resnet and mobilenet, will run the test only once and populate the test results in both resnet and mobilenet

[nvukobratTT]

Great work @chandrasekaranpradeep! I don't see any major issues from code changes. However, I would like to confirm few details before proceeding with further review. That said, can you provide details about achieved results with this change?

Here are a few questions as a guideline:

1. How long tests are executing now, vs before this change? Please provide details of machine on which you run this.
2. Summary of workflow changes? E.g. how generation and run work now (generate unique across all models; run only unique and populate details for all variants)
3. Do we have ability to separate test generation and test runs? E.g. to have a workflow where we generate these tests on weekly, but running them on nightly.
4. What is the process of including generated op tests as part of the push pipeline?
5. More detail on how params/constants form model are handled? Do we rely on random tensors now, and can this be swapped easily per need? E.g. debugging PCC issues with original weights instead of ones generated in the runtime.

Other:

1. @vmilosevic is it possible to have workflow run when the specific part of the code has changed? @chandrasekaranpradeep for example when TVM ref change, to run job to update these generated unique op tests.

[chandrasekaranpradeep]

@nvukobratTT

1. How long tests are executing now, vs before this change? Please provide details of machine on which you run this.

I have checked the behaviour of the implementation but haven't noted the execution time before and after the change, will check and let you know here.

Summary of workflow changes? E.g. how generation and run work now (generate unique across all models; run only unique and populate details for all variants)

Summary of previous workflow:

1. Generate the unique op tests for the models present in the directory path specified by the user(i. e forge/test/models/pytorch):
   1. Collected all the pytest present in the specified directory path
   2. Run the pytest and extract unique op configuration and generate forge module with test functions and additionally export the unique op config test details as excel.
2. Run the generated unique op config tests and generate markdowns
   1. Run the generated unique op config forge test cases model wise and generate sub markdown files contains unique operation configurations, pass/fail status for each compiler component, and failure reasons.
   2. After running all the models unique op test, generate root markdown file for summarising the result for all the models.

Summary of the new workflow:

1. Generate the unique op tests for the models present in the directory path specified by the user(i. e forge/test/models/pytorch):
   1. Collecting and running the pytest and extracting unique op config and generating forge module is similar to the previous workflow but in addition to exporting unique op config test details as excel file, will export the model metadata information which includes module_name, framework name, model params and buffers file path which will be used for extracting unique op configuration across all the models.
2. Run the generated unique op config tests and generate markdowns.
   1. Instead of running the generated unique op config forge test cases model wise, by utilising the same UniqueOperations class(i.e which is used for extracting unique op config from model), will extract unique op configuration across all the models and run the extracted unique op configuration across all the models and populate the test result to the respective models.
   2. After running extracted unique op config across all the models will generate sub markdown and root markdown files.

3. Do we have ability to separate test generation and test runs? E.g. to have a workflow where we generate these tests on weekly, but running them on nightly.
4. What is the process of including generated op tests as part of the push pipeline?
5. More detail on how params/constants form model are handled? Do we rely on random tensors now, and can this be swapped easily per need? E.g. debugging PCC issues with original weights instead of ones generated in the runtime.

All these question are related to generation of test for nightly from the extracted unique op configuration across all the models, the PR doesn't have implementation for generating test for nightly from extracted unique op configuration across all the models.

The pchandrasekaran/generate_nightly_test branch that I am working has the changes for generating test for nightly which will be later include as part of push pipeline

The idea is to generate nightly test by using the exported unique op config test details excel and models metadata info json files produced from the Generate the unique op tests for the models present in the directory path specified by the user(i. e forge/test/models/pytorch) step in the new workflow

In the model analysis script main function, I have added an option to generate nightly test.

When we enable the option of generating nightly test, after generating unique op test from all the models with xlsx and json files, we are extracting unique op configuration across all the models and then generate nightly test for extracted unique op configuration across models.

Instead of loading actual parameters and constants from models with process_framework_paremeter method, now the parameters and constants are randomly generated by modifying the add_constants method in ForgeModule in forge/forge/module.py and Parameter class in forge/forge/parameter.py and also done some modification in the write_class_defination function in ForgeWritter.

Added recorded_property fixtures in the nightly test function which includes model_name, variant_name, framework_op_name and op_kind

In sum, the model_analysis script will additional generate nightly test with model_analysis_nightly marker and then create a separate PR for the generate nightly test and then we will have some github workflow to run the generated nightly based upon the test results we can remove the failed test cases from the same PR branch and check it and merge it into main.

Currently comparing the generated nightly test result with unique op test result for bert models which also includes pcc drop check when we use random tensor instead of actual params and constants from the model. The idea is the unique op and generated nightly test results should match.

Whats your thoughts on this nikola?

[chandrasekaranpradeep]

Genereted nightly test from model analysis script:
nightly.zip

[chandrasekaranpradeep]

@nvukobratTT I have tested the model analysis script before and after optimisation with the forge/test/models/pytorch/text/albert/test_albert.py::test_albert_masked_lm_pytorch test cases which contains 8 variants and below is the results.

Model analysis script overall execution time

   Before optimisation   :   9568 Seconds (i.e 2 Hours, 39 Minutes and 27 Seconds)
   After optimisation    :   3852 Seconds (i.e 1 Hour, 4 Minutes and 12 Seconds)

[nvukobratTT]

Thanks for the details! Sounds good!

The pchandrasekaran/generate_nightly_test branch that I am working has the changes for generating test for nightly which will be later include as part of push pipeline

Think about how we can include passing ones right away as part of push pipeline, while keeping other part of nightly. Ofc, this is more question for upcoming PR rather then this one. For other details related to futhre PR, let's discuss in our sync tomorrow.