[VOQ][saidump] Add saidump unit test scripts #3079
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, sonic-net#1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null sonic-net#2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" sonic-net#3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null sonic-net#4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" sonic-net#5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh
• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh
• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh
• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh
sonic-net#2972 SAI DUMP based on the route table size * [saidump] • Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2, install Python library rdbtools into the syncd containter. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Updated sonic-buildimage/build_debian.sh, to add a new script file: files/scripts/saidump.sh into the host. This shell file does the below steps: For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files docker exec syncd$1 sh -c "rdb --command json /var/run/redis$1/dump.rdb | tee /var/run/redis$1/dump.json > /dev/null" 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json -m 100" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, to check the asic db size and if it is larger than xxx entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB.
…_LIMIT_DIRECT_ITERATION to 24000
To add the rdbtools into base docker. Move saidump.sh from host to syncd docker container.
|
@judyjoseph , @mlok-nokia , please help to review this PR. Thanks. |
JunhongMao
added a commit
to JunhongMao/sonic-utilities
that referenced
this pull request
Dec 13, 2023
sonic-net#2972 added two below functions into scripts/generate_dump. get_route_table_size_by_asic_id_and_ipver save_saidump_by_route_size The unittest scripts need to be added. Related PRs: sonic-net#2972 sonic-net/sonic-buildimage#16466 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 Microsoft ADO (25892277): Add two scripts: tests/saidump_test.py tests/saidump_test.sh To use below 6 test cases to verify the functionality of get_route_table_size_by_asic_id_and_ipver and save_saidump_by_route_size behave correctly. ``` saidump test list format: [ACIS number, ipv4 and ipv6 route table size, expected function save_cmd arguments] saidump_test_list = [ [1, 10000, "docker exec syncd saidump saidump"], [1, 12000, "docker exec syncd saidump saidump"], [1, 12001, "docker exec syncd saidump.sh saidump"], [1, 20000, "docker exec syncd saidump.sh saidump"], [2, 10000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12001, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"], [2, 20000, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"] ] ``` During the compiling stage, run the below command to check if it's PASSED. jumao@1b1ffba5949a:/sonic/src/sonic-utilities$ time python3 setup.py test tests/saidump_test.py::test_saidump PASSED
sonic-net#2972 added two below functions into scripts/generate_dump. get_route_table_size_by_asic_id_and_ipver save_saidump_by_route_size The unittest scripts need to be added. Related PRs: sonic-net#2972 sonic-net/sonic-buildimage#16466 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 Microsoft ADO (25892277): Add two scripts: tests/saidump_test.py tests/saidump_test.sh To use below 6 test cases to verify the functionality of get_route_table_size_by_asic_id_and_ipver and save_saidump_by_route_size behave correctly. ``` saidump test list format: [ACIS number, ipv4 and ipv6 route table size, expected function save_cmd arguments] saidump_test_list = [ [1, 10000, "docker exec syncd saidump saidump"], [1, 12000, "docker exec syncd saidump saidump"], [1, 12001, "docker exec syncd saidump.sh saidump"], [1, 20000, "docker exec syncd saidump.sh saidump"], [2, 10000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12001, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"], [2, 20000, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"] ] ``` During the compiling stage, run the below command to check if it's PASSED. jumao@1b1ffba5949a:/sonic/src/sonic-utilities$ time python3 setup.py test tests/saidump_test.py::test_saidump PASSED
|
@judyjoseph , @mlok-nokia , please help to review this PR. Thanks. |
JunhongMao
added a commit
to JunhongMao/sonic-utilities
that referenced
this pull request
Apr 23, 2024
This reverts commit 1322095.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
#2972 added two below functions into scripts/generate_dump.
get_route_table_size_by_asic_id_and_ipver
save_saidump_by_route_size
The unittest scripts need to be added according to the below comments:
#2972 (comment)
Related PRs:
#2972
sonic-net/sonic-buildimage#16466
sonic-net/sonic-sairedis#1288
sonic-net/sonic-sairedis#1298
Work item tracking
Microsoft ADO (25892277):
How I did it
Add two scripts:
tests/saidump_test.py
tests/saidump_test.sh
To use below 6 test cases to verify the functionality of get_route_table_size_by_asic_id_and_ipver and save_saidump_by_route_size behave correctly.
How to verify it
During the compiling stage, run the below command to check if it's PASSED.
jumao@1b1ffba5949a:/sonic/src/sonic-utilities$ time python3 setup.py test
tests/saidump_test.py::test_saidump PASSED
Previous command output (if the output of a command-line utility has changed)
New command output (if the output of a command-line utility has changed)