After switching OOD stop Dec App get prompt of “Can not stop app. app status not found in list“

[Mikestar01]

Describe the bug
After switching OOD, I coudn't stop all the Dec Apps(in Cyber Chat or CYFS Browser)， get the prompt of “Can not stop app. app status not found in list“

System information
the tested version:
services：v1.1.1.82
Cyber Chat: Android 1.0.49（beta）

[lurenpluto]

Switching OOD is a complex operation with many steps, the most important of which is data synchronization.
The problem seems to be that the data synchronization of the active OOD is not completed, or there is a problem with the synchronization, and then the OOD is switched, resulting in some key data such as local-status of the app-manager not being found on the new OOD

So there are a few things to confirm:

1. Was the data already synchronized when you switched the OOD?
2. Was the app already installed and started on the active OOD before the switch? (make sure that local-status data already exists)

[lurenpluto]

Describe the bug After switching OOD, I coudn't stop all the Dec Apps(in Cyber Chat or CYFS Browser)， get the prompt of “Can not stop app. app status not found in list“

System information the tested version: services：v1.1.1.82 Cyber Chat: Android 1.0.49（beta）

Can you attach the logs of active OOD and standby OOD, including gateway and app-manager services, for diagnosing the problem?

[Mikestar01]

Switching OOD is a complex operation with many steps, the most important of which is data synchronization. The problem seems to be that the data synchronization of the active OOD is not completed, or there is a problem with the synchronization, and then the OOD is switched, resulting in some key data such as local-status of the app-manager not being found on the new OOD

So there are a few things to confirm:

1. Was the data already synchronized when you switched the OOD?
2. Was the app already installed and started on the active OOD before the switch? (make sure that local-status data already exists)

1.the process was normal while switched the OOD in Cyber Chat
2.the answer is yes.

[weiqiushi]

ood-daemon_1908_rCURRENT.log
app-manager_6008_rCURRENT.log
gateway_2196_rCURRENT.log
I got some log files from @Mikestar01 's OOD，it shows:

2023-04-18 15:13:56 gateway on OOD started
2023-04-18 15:15:55 ood-daemon finds itself switching to primary OOD, starts other service processes, and restarts gateway, in order to get gateway into primary OOD state
2023-04-18 15:15:58.039 app-manager on OOD starts, connects to the gateway process, and starts initializing the internal state
2023-04-18 15:15:58.263 gateway is restarted, app-manager is in the process of initialization

Since the gateway is not present when the app-manager is reading the data needed for initialization from the gateway, the app-manager assumes that the data acquisition has failed and resets itself to the initial state without any app installed.

After that, the user connects to the rebooted gateway through the App Management page of CYFS Browser, and gets the correct list of apps, and sends commands to AppManager to control these apps

When AppManager receives the command to control the apps, the local state is not synchronized with the stack state, which causes app-manager to consider the command invalid and return an error

[weiqiushi]

This is related to some historical issues:

When app-manager was developed and running online, the CYFS stack did not have root-state as a data access model, and app-manager managed data internally and interacted with the outside separately.

Later, when the CYFS stack developed the root-state access model, the app-manager and the DecApp management page of CYFS Browser started to interact with data through the root-state, which brought some problems in state synchronization

A fundamental solution here is to refactor the state management logic of app-manager to fully rely on root-state, and this change will be implemented in the next few releases.

[weiqiushi]

This may have a simple solution: when AppManager connects to the CYFS stack, check the state of the stack. If the stack is still in the Standby-OOD state, then it exits itself and waits until the next time it is pulled up until the stack is in the master OOD state

[lizhihongTest]

@weiqiushi When verifying this problem, I found that after switching from OOD, starting DEC APP from OOD is executed serially. This is a relatively long operation to start all DEC APPs. Can it be optimized to start in parallel? There may be many DEC APPs installed by users

[lizhihongTest]

@tracy101 After switching OOD,The dec app can start successfuly after version Nighlty 1.1.0.756 and after version Beta 1.1.1.83