Fixing service-restart testcases.

[rraghav-cisco]

Description of PR

Summary:
The pfcwd_basic service-restart cases keep failing due to: sonic-net/sonic-buildimage#20637

The ask is not to restart swss multiple times without doing a config reload in between. So in this PR:

1. we are doing config-reload for every iteration of the test
2. The swss restart is done only once in one DUT.
3. The asic is randomly picked, and the swss of that ASIC is restarted instead of doing the restart for all asics.
4. Also added checks to make sure the services, interfaces and bgp are up before proceding with the ixia traffic.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 202012
• 202205
• 202305
• 202311
• 202405

Approach

What is the motivation for this PR?

The issue: sonic-net/sonic-buildimage#20637

How did you do it?

Pls see the description.

How did you verify/test it?

Ran it on my TB.

=========================================================================================================================== PASSES ===========================================================================================================================
____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] _____________________________________________________________________________________
____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] ____________________________________________________________________________________
____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] _____________________________________________________________________________________
____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ____________________________________________________________________________________
----------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service/2024-11-14-00-05-11/tr_2024-11-14-00-05-11.xml ------------------------------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------------------------------------------------------------------- live log sessionfinish -------------------------------------------------------------------------------------------------------------------
01:31:34 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================================== short test summary info ===================================================================================================================
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss]
========================================================================================================= 4 passed, 7 warnings in 5180.68s (1:26:20) =========================================================================================================
sonic@ixia-sonic-mgmt-whitebox:/data/tests$

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================================================================================================== PASSES ===========================================================================================================================
____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] _____________________________________________________________________________________
____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] ____________________________________________________________________________________
____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] _____________________________________________________________________________________
____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ____________________________________________________________________________________
---------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service-2/2024-11-14-02-47-47/tr_2024-11-14-02-47-47.xml -----------------------------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------------------------------------------------------------------- live log sessionfinish -------------------------------------------------------------------------------------------------------------------
04:14:03 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================================== short test summary info ===================================================================================================================
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss]
========================================================================================================= 4 passed, 7 warnings in 5173.22s (1:26:13) =========================================================================================================
sonic@ixia-sonic-mgmt-whitebox:/data/tests$
=========================================================================================================================== PASSES ===========================================================================================================================
____________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss] ____________________________________________________________________________________
___________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss] ____________________________________________________________________________________
____________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss] ____________________________________________________________________________________
___________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss] ____________________________________________________________________________________
---------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service-2/2024-11-14-06-39-15/tr_2024-11-14-06-39-15.xml -----------------------------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------------------------------------------------------------------- live log sessionfinish -------------------------------------------------------------------------------------------------------------------
08:10:42 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================================== short test summary info ===================================================================================================================
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss]
PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss]
========================================================================================================= 4 passed, 7 warnings in 5484.86s (1:31:24) =========================================================================================================
sonic@ixia-sonic-mgmt-whitebox:/data/tests$ 

[rraghav-cisco]

@sdszhang , @amitpawar12 , @selldinesh : FYI.