Backup STATE_DB PORT_TABLE|Ethernet during warm-reboot #3111

mihirpat1 · 2024-01-05T03:07:16Z

What I did

Currently, entire PORT_TABLE in STATE_DB is being deleted during warm-reboot. Due to this, host_tx_ready changes to false after warm-reboot which causes the link to remain down.

How I did it

Backing up host_tx_ready, NPU_SI_SETTINGS_SYNC_STATUS and CMIS_REINIT_REQUIRED fields from `STATE_DB PORT_TABLE* during warm-reboot now.

How to verify it

Verified that host_tx_ready in STATE_DB PORT_TABLE is retained after warm-reboot and the link remains up. Also, ensured that the keys CMIS_REINIT_REQUIRED and NPU_SI_SETTINGS_SYNC_STATUS are retained after warm-reboot.
Before warm-reboot

root@sonic:/home/admin# redis-cli -n 6 hgetall "PORT_TABLE|Ethernet0"
 1) "state"
 2) "ok"
 3) "netdev_oper_status"
 4) "up"
 5) "admin_status"
 6) "up"
 7) "mtu"
 8) "9100"
 9) "CMIS_REINIT_REQUIRED"
10) "false"
11) "NPU_SI_SETTINGS_SYNC_STATUS"
12) "NPU_SI_SETTINGS_DEFAULT"
13) "supported_speeds"
14) "40000,100000"
15) "supported_fecs"
16) "none,rs"
17) "host_tx_ready"
18) "true"
19) "speed"
20) "100000"
21) "fec"
22) "N/A"
root@sonic:/home/admin#

After warm-reboot script backs up PORT_TABLE and deletes unwanted fields

root@sonic:/home/admin# redis-cli -n 6 hgetall "PORT_TABLE|Ethernet0"
1) "CMIS_REINIT_REQUIRED"
2) "false"
3) "NPU_SI_SETTINGS_SYNC_STATUS"
4) "NPU_SI_SETTINGS_DEFAULT"
5) "host_tx_ready"
6) "true"
root@sonic:/home/admin#

After switch boot-up post warm-reboot

root@sonic:/home/admin# redis-cli -n 6 hgetall "PORT_TABLE|Ethernet0"
 1) "state"
 2) "ok"
 3) "netdev_oper_status"
 4) "up"
 5) "admin_status"
 6) "up"
 7) "mtu"
 8) "9100"
 9) "supported_speeds"
10) "40000,100000"
11) "supported_fecs"
12) "none,rs"
13) "CMIS_REINIT_REQUIRED"
14) "false"
15) "NPU_SI_SETTINGS_SYNC_STATUS"
16) "NPU_SI_SETTINGS_DEFAULT"
17) "host_tx_ready"
18) "true"
19) "speed"
20) "100000"
21) "fec"
22) "N/A"
root@sonic:/home/admin#

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Signed-off-by: Mihir Patel <patelmi@microsoft.com>

…eting unwanted fields during warm-reboot

vaibhavhd · 2024-01-19T19:37:49Z

scripts/fast-reboot

-            if not string.match(k, 'FDB_TABLE|') and not string.match(k, 'WARM_RESTART_TABLE|') \
+            if string.match(k, 'PORT_TABLE|Ethernet') then
+                for i, f in ipairs(redis.call('hgetall', k)) do
+                    if i % 2 == 1 then


I am not clear on what this check looking for - if i % 2 == 1

@vaibhavhd - This logic will help in selecting the field from FieldValue Pair.

Since the getall command will return a flat list of field followed by its corresponding value, the above logic will help in selecting the field and delete the corresponding FieldValue Pair

Below snippet has a total of 11 FieldValue pairs.

root@sonic:/home/admin# redis-cli -n 6 hgetall "PORT_TABLE|Ethernet0" 1) "state" 2) "ok" 3) "netdev_oper_status" 4) "up" 5) "admin_status" 6) "up" 7) "mtu" 8) "9100" 9) "CMIS_REINIT_REQUIRED" 10) "false" 11) "NPU_SI_SETTINGS_SYNC_STATUS" 12) "NPU_SI_SETTINGS_DEFAULT" 13) "supported_speeds" 14) "40000,100000" 15) "supported_fecs" 16) "none,rs" 17) "host_tx_ready" 18) "true" 19) "speed" 20) "100000" 21) "fec" 22) "N/A" root@sonic:/home/admin#

vaibhavhd · 2024-01-19T19:41:29Z

scripts/fast-reboot

@@ -247,7 +247,17 @@ function backup_database()
    # Delete keys in stateDB except FDB_TABLE|*, MIRROR_SESSION_TABLE|*, WARM_RESTART_ENABLE_TABLE|*, FG_ROUTE_TABLE|*
    sonic-db-cli STATE_DB eval "
        for _, k in ipairs(redis.call('keys', '*')) do
-            if not string.match(k, 'FDB_TABLE|') and not string.match(k, 'WARM_RESTART_TABLE|') \
+            if string.match(k, 'PORT_TABLE|Ethernet') then


@zjswhhh do you think keeping port_table intact in state-db will cause any change in mux init logic for dualtor warmboot?

No it shouldn't.

vaibhavhd · 2024-01-19T19:41:58Z

scripts/fast-reboot

@@ -247,7 +247,17 @@ function backup_database()
    # Delete keys in stateDB except FDB_TABLE|*, MIRROR_SESSION_TABLE|*, WARM_RESTART_ENABLE_TABLE|*, FG_ROUTE_TABLE|*


Please update the comment accordingly.

vaibhavhd · 2024-01-19T19:55:52Z

scripts/fast-reboot

+                        if not string.match(f, 'host_tx_ready') \
+                            and not string.match(f, 'NPU_SI_SETTINGS_SYNC_STATUS') \
+                            and not string.match(f, 'CMIS_REINIT_REQUIRED') then


From the PR's description it sounds like you need only host_tx_ready field. What is the reason for also persisting NPU and CMIS fields? Please add the reasoning as a comment here too.

Have you considered cross branch and in-branch warm-reboots where the 3 fields might get modified (deleted, name change, etc?) in the target image? How will the target image handle a scenario when state db has some filed which is not supported. This question arises from the fact that you are not preserving entire table, but just 3 fields from it. Existing cases preserved entire table.

To prevent us from getting into any unlikely scenarios in # 2 above, this handling can be done by the target image's portorch. In other words, when the device boots into target image, the portsorch initializes these fields afresh after checking that system is undergoing warmboot by checking system warm-restart flag. The downside in this approach is that portorch will have no idea what these fields were set to in the base image.

The argument is same for other fields such as OPER state (netdev_oper_status) and speeds. In the recovery path of warmboot these fields are set afresh based on SAI get calls (I think)

@vaibhavhd you may want to review this https://github.com/sonic-net/sonic-utilities/pull/3240/files

longhuan-cisco · 2024-02-02T02:11:39Z

Just curious, what was the reason to not back up the entire table? Is it because some of the fields (e.g. netdev_oper_status ) should be re-populated after warm-reboot?

mihirpat1 · 2024-02-02T02:45:59Z

Just curious, what was the reason to not back up the entire table? Is it because some of the fields (e.g. netdev_oper_status ) should be re-populated after warm-reboot?

@longhuan-cisco - Yes, you are correct. Hence, we decided to preserve selected fields which xcvrd/OA cares about and delete other fields from STATE_DB.

longhuan-cisco · 2024-05-30T22:14:12Z

As discussed, I tested the change from this PR, host_tx_ready gets retained properly after warm-reboot and link stays up (especially for those CMIS modules).

@mihirpat1 @prgeor Could you please continue on this PR for the remaining?

root@t0-dut:/home/cisco# show reboot-cause history
Name                 Cause        Time                             User    Comment
-------------------  -----------  -------------------------------  ------  ---------
2024_05_22_07_52_46  warm-reboot  Wed May 22 07:49:46 UTC 2024     cisco   N/A
...

May 22 07:55:26.154419 cmono-t0-dut NOTICE pmon#xcvrd[27]: XCVRD INIT: Wait for port config is done
May 22 07:55:26.156638 cmono-t0-dut NOTICE pmon#xcvrd[27]: XCVRD INIT: After port config is done
May 22 07:55:26.183632 cmono-t0-dut NOTICE pmon#xcvrd[27]: Start daemon main loop with thread count 3
May 22 07:55:26.183632 cmono-t0-dut NOTICE pmon#xcvrd[27]: Started thread CmisManagerTask
May 22 07:55:26.183675 cmono-t0-dut NOTICE pmon#xcvrd[27]: Started thread DomInfoUpdateTask
May 22 07:55:26.183675 cmono-t0-dut NOTICE pmon#xcvrd[27]: Started thread SfpStateUpdateTask
...
May 22 07:55:26.198509 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet32 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'true', 'state': 'ok', 'netdev_oper_status': 'up', 'admin_status': 'up', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs', 'speed': '400000'}
May 22 07:55:26.198532 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet56 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'true', 'state': 'ok', 'netdev_oper_status': 'up', 'admin_status': 'up', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs', 'speed': '400000'}
May 22 07:55:26.198554 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet0 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'false', 'state': 'ok', 'netdev_oper_status': 'down', 'admin_status': 'down', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs'}
May 22 07:55:26.198577 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet16 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'true', 'state': 'ok', 'netdev_oper_status': 'up', 'admin_status': 'up', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs', 'speed': '400000'}
May 22 07:55:26.198601 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet128 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'true', 'state': 'ok', 'netdev_oper_status': 'up', 'admin_status': 'up', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs', 'speed': '400000'}
May 22 07:55:26.198618 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet72 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'true', 'state': 'ok', 'netdev_oper_status': 'up', 'admin_status': 'up', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs', 'speed': '400000'}
May 22 07:55:26.198643 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet120 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'true', 'state': 'ok', 'netdev_oper_status': 'up', 'admin_status': 'up', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs', 'speed': '400000'}
May 22 07:55:26.198661 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet192 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'true', 'state': 'ok', 'netdev_oper_status': 'up', 'admin_status': 'up', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs', 'speed': '400000'}
May 22 07:55:26.198689 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet200 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'false', 'state': 'ok', 'netdev_oper_status': 'down', 'admin_status': 'down', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs'}
May 22 07:55:26.198712 cmono-t0-dut WARNING pmon#xcvrd[27]: $$$ Ethernet176 handle_port_update_event() : op=SET DB:STATE_DB Table:PORT_TABLE fvp {'host_tx_ready': 'true', 'state': 'ok', 'netdev_oper_status': 'up', 'admin_status': 'up', 'mtu': '9100', 'supported_speeds': '200000,400000', 'supported_fecs': 'rs', 'speed': '400000'}
...

prgeor · 2024-05-30T23:37:30Z

@StormLiangMS @yxieca @bingwang-ms please cherry pick this to 202311. Need for warm reboot support for platforms using CMIS optics

* Backup STATE_DB PORT_TABLE during warm-reboot Signed-off-by: Mihir Patel <patelmi@microsoft.com> * Backing up selected fields from STATE_DB PORT_TABLE|Ethernet* and deleting unwanted fields during warm-reboot --------- Signed-off-by: Mihir Patel <patelmi@microsoft.com>

mssonicbld · 2024-06-03T16:25:04Z

Cherry-pick PR to 202311: #3352

* Backup STATE_DB PORT_TABLE during warm-reboot Signed-off-by: Mihir Patel <patelmi@microsoft.com> * Backing up selected fields from STATE_DB PORT_TABLE|Ethernet* and deleting unwanted fields during warm-reboot --------- Signed-off-by: Mihir Patel <patelmi@microsoft.com>

prgeor · 2024-06-18T23:10:32Z

@bingwang-ms we need this in 202405

bingwang-ms · 2024-06-18T23:18:49Z

@prgeor Seems there is cherry-pick conflict. Please double check

mihirpat1 · 2024-06-18T23:30:53Z

@prgeor Seems there is cherry-pick conflict. Please double check

@bingwang-ms I have removed the 202405 tags since this is already part of 202405.

Backup STATE_DB PORT_TABLE during warm-reboot

1c51a5f

Signed-off-by: Mihir Patel <patelmi@microsoft.com>

mihirpat1 requested a review from prgeor January 5, 2024 03:07

This was referenced Jan 5, 2024

OA crash handling to reinitialize port through xcvrd sonic-net/SONiC#1432

Merged

Issuing DEL command to remove APPL_DB:PORT_TABLE sonic-net/sonic-buildimage#17065

Closed

Backing up selected fields from STATE_DB PORT_TABLE|Ethernet* and del…

5cb4fca

…eting unwanted fields during warm-reboot

mihirpat1 changed the title ~~Backup STATE_DB PORT_TABLE during warm-reboot~~ Backup STATE_DB PORT_TABLE|Ethernet* during warm-reboot Jan 11, 2024

mihirpat1 changed the title ~~Backup STATE_DB PORT_TABLE|Ethernet* during warm-reboot~~ Backup STATE_DB PORT_TABLE|Ethernet during warm-reboot Jan 11, 2024

mihirpat1 requested a review from vaibhavhd January 11, 2024 17:31

vaibhavhd reviewed Jan 19, 2024

View reviewed changes

prgeor approved these changes May 30, 2024

View reviewed changes

prgeor merged commit a9720bf into sonic-net:master May 30, 2024
5 checks passed

prgeor added the Request for 202311 Branch label May 30, 2024

yxieca added the Approved for 202311 Branch label Jun 3, 2024

mssonicbld mentioned this pull request Jun 3, 2024

[action] [PR:3111] Backup STATE_DB PORT_TABLE|Ethernet during warm-reboot (#3111) #3352

Merged

mssonicbld added the Created PR to 202311 Branch label Jun 3, 2024

mssonicbld added Included in 202311 Branch and removed Created PR to 202311 Branch labels Jun 3, 2024

prgeor added the Request for 202405 Branch label Jun 18, 2024

mssonicbld added the Cherry Pick Conflict_202405 label Jun 18, 2024

bingwang-ms added Approved for 202405 Branch and removed Cherry Pick Conflict_202405 labels Jun 18, 2024

mssonicbld added the Cherry Pick Conflict_202405 label Jun 18, 2024

mihirpat1 removed Request for 202405 Branch Approved for 202405 Branch Cherry Pick Conflict_202405 labels Jun 18, 2024

PeterTSW-EC mentioned this pull request Oct 7, 2024

Cherry-pick the community PR #3111 "Backup STATE_DB PORT_TABLE|Ethernet during warm-reboot". edge-core/sonic-utilities#34

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backup STATE_DB PORT_TABLE|Ethernet during warm-reboot #3111

Backup STATE_DB PORT_TABLE|Ethernet during warm-reboot #3111

mihirpat1 commented Jan 5, 2024 •

edited

Loading

vaibhavhd Jan 19, 2024

mihirpat1 Jan 30, 2024

vaibhavhd Jan 19, 2024

zjswhhh Mar 26, 2024

vaibhavhd Jan 19, 2024

vaibhavhd Jan 19, 2024

prgeor Mar 25, 2024

longhuan-cisco commented Feb 2, 2024

mihirpat1 commented Feb 2, 2024

longhuan-cisco commented May 30, 2024

prgeor commented May 30, 2024

mssonicbld commented Jun 3, 2024

prgeor commented Jun 18, 2024

bingwang-ms commented Jun 18, 2024

mihirpat1 commented Jun 18, 2024

		@@ -247,7 +247,17 @@ function backup_database()
		# Delete keys in stateDB except FDB_TABLE\|, MIRROR_SESSION_TABLE\|, WARM_RESTART_ENABLE_TABLE\|, FG_ROUTE_TABLE\|

Backup STATE_DB PORT_TABLE|Ethernet during warm-reboot #3111

Backup STATE_DB PORT_TABLE|Ethernet during warm-reboot #3111

Conversation

mihirpat1 commented Jan 5, 2024 • edited Loading

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

vaibhavhd Jan 19, 2024

Choose a reason for hiding this comment

mihirpat1 Jan 30, 2024

Choose a reason for hiding this comment

vaibhavhd Jan 19, 2024

Choose a reason for hiding this comment

zjswhhh Mar 26, 2024

Choose a reason for hiding this comment

vaibhavhd Jan 19, 2024

Choose a reason for hiding this comment

vaibhavhd Jan 19, 2024

Choose a reason for hiding this comment

prgeor Mar 25, 2024

Choose a reason for hiding this comment

longhuan-cisco commented Feb 2, 2024

mihirpat1 commented Feb 2, 2024

longhuan-cisco commented May 30, 2024

prgeor commented May 30, 2024

mssonicbld commented Jun 3, 2024

prgeor commented Jun 18, 2024

bingwang-ms commented Jun 18, 2024

mihirpat1 commented Jun 18, 2024

mihirpat1 commented Jan 5, 2024 •

edited

Loading