[warm-reboot] Upgrade from 201911 to 202205 is failing due to fpmsyncd crash

[volodymyrsamotiy]

Description

201911->202205 warm-upgrade is failing because of fpmsyncd crash on reconciliation stage during 202205 initialization after warm-reboot.

Nov  7 14:18:00.088021 sonic NOTICE bgp#fpmsyncd: :- main: Warm-Restart timer expired.
Nov  7 14:18:00.088507 sonic NOTICE bgp#fpmsyncd: :- reconcile: Warm-Restart: Initiating reconciliation process for bgp application.
Nov  7 14:18:00.089566 sonic INFO bgp#supervisord: fpmsyncd fpmsyncd: ../warmrestart/warmRestartHelper.cpp:295: bool swss::WarmStartHelper::compareAllFV(const std::vector<std::pair<std::__cxx11::basic_string<char>, std::__cxx
11::basic_string<char> > >&, const std::vector<std::pair<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> > >&): Assertion `v1Iter != v1Map.end()' failed.
Nov  7 14:18:01.545656 sonic INFO bgp#supervisord 2022-11-03 14:18:01,544 INFO exited: fpmsyncd (terminated by SIGABRT (core dumped); not expected)
Nov  7 14:18:02.307563 sonic INFO systemd[1]: run-docker-runtime\x2drunc-moby-216ca677eef9bbee9b1202cc3b1ddbff50a8ce9b1330da2f3fa1ea3e9afa6364-runc.F02N9R.mount: Succeeded.
Nov  7 14:18:02.580120 sonic INFO bgp#supervisor-proc-exit-listener: Process 'fpmsyncd' exited unexpectedly. Terminating supervisor 'bgp'
Nov  7 14:18:02.583203 sonic INFO bgp#supervisord 2022-11-07 14:18:02,580 WARN received SIGTERM indicating exit request
Nov  7 14:18:02.583414 sonic INFO bgp#supervisord 2022-11-07 14:18:02,580 INFO waiting for supervisor-proc-exit-listener, rsyslogd, staticd, zebra, bgpd, bgpcfgd, bgpmon to die
Nov  7 14:18:03.585041 sonic INFO bgp#supervisord 2022-11-07 14:18:03,584 INFO stopped: bgpmon (terminated by SIGTERM)
Nov  7 14:18:04.595653 sonic INFO bgp#supervisord 2022-11-07 14:18:04,594 INFO stopped: bgpcfgd (exit status 0)
Nov  7 14:18:04.611052 sonic INFO bgp#supervisord 2022-11-07 14:18:04,610 INFO stopped: bgpd (terminated by SIGKILL)

fpmsyncd is crashing due to shown below assert during routes reconciliation.
https://github.com/sonic-net/sonic-swss/blob/202205/warmrestart/warmRestartHelper.cpp#L295

/*
 * Compare all field-value-tuples within two vectors.
 *
 * Example: v1 {nexthop: 10.1.1.1, ifname: eth1}
 *          v2 {nexthop: 10.1.1.2, ifname: eth2}
...
bool WarmStartHelper::compareAllFV(const std::vector<FieldValueTuple> &v1,
                                   const std::vector<FieldValueTuple> &v2)
{
 ...
        assert(v1Iter != v1Map.end());
...}

Not 100% sure if it is root cause for fpmsyncd crash, but it looks like route entries look different in 201911 and 202205. Reconciliation logic goes through all the routes in APP_DB, compares all the attributes and expects them to match. But there is additional "weight"" attribute in 202205, which is not present in 201911. As a result fpmsyncd is aborted.

Example of APP_DD route config in 201911:

127.0.0.1:6379> HGETALL "ROUTE_TABLE:192.237.160.128/25"
1) "nexthop"
2) "10.0.0.57,10.0.0.59,10.0.0.61,10.0.0.63"
3) "ifname"
4) "PortChannel101,PortChannel102,PortChannel103,PortChannel104"

Example of APP_DD route config in 202205:

127.0.0.1:6379> HGETALL "ROUTE_TABLE:192.206.96.0/25"
1) "nexthop"
2) "10.0.0.57,10.0.0.59,10.0.0.61,10.0.0.63"
3) "ifname"
4) "PortChannel101,PortChannel102,PortChannel103,PortChannel104"
5) "weight"
6) "1,1,1,1"

Steps to reproduce the issue:

1. Deploy 201911 image and T0 topology
2. Install 202205 image
3. Perform war-reboot
4. Check syslog already on 202205 image and observe that fpmsyncd crashed on reconciliation stage

Describe the results you received:

fpmsyncd crash after warm-upgrading from 201911 to 202205

Describe the results you expected:

No fpmsyncd crash after upgrading from 201911 to 202205

Output of show version:

• Latest 201911 image (e.g. image built on top of 7781399bb hash was used)
• Latest 202205 image (e.g. image built on top of d2540ae31 hash was used)

Output of show techsupport:

N/A

Additional information you deem important (e.g. issue happens only occasionally):

N/A

[nkelapur]

The old design for warm-reboot was expecting that the number of fields will be same before and after warmreboot. However for a warm-reboot with upgrade scenario clearly this is not the case. Hence the assert needs to be removed and instead need to return "true" so that the new route entry gets written to the APP-DB

[vaibhavhd]

@adyeung , do you have any update on this issue?

[adyeung]

@vaibhavhd expecting a fix to be posted by nkelapur

[nkelapur]

created a commit sonic-net/sonic-swss#2544 with the fix