Rework ws-agent activity notification mechanism

[akorneta]

Description

In cases when notification scheduler does not notify about activity till the current time >= last update time + threshold then the activity notification will be sent in the same thread in which this activity occurred. Such behaviour may lead to problems because the request for activity has default timeout 1 minute, and if the master is not available for this period it will block the activity thread for one minute. The activity thread its tomcat thread where LastAccessTimeFilter call WorkspaceActivityNotifier.onActivity(), and during this period the client won't receive a response from the agent.

The problem is not abstract, it occurs while recovery on docker infrastructure.

• on post-construct ws-master recovers runtime
• then checks servers with request timeout 3 sec
• server checks produce activity and wait for 3 sec till agent notifies about activity master(master) in the same thread after timeout is exceeded server checks fail and workspaces considered as stopped.

[artaleks9]

Still actual for ver.6.5.0