Relay Node draining logic

[maxwnewcomer]

Draining logic

I believe that the following draining logic for a worker will work.

1. Room status in UP
2. SIGINT to Relay node
3. Redis update DRAINING status with some TTL for each room
4. Spawn draining worker to keep status up to date
5. Deny New connection
6. Drop current connections
   • Update relay takeover logic to not takeover on valid TTL DRAINING status
7. Start potential room persistence for all rooms
8. Once room persistence done set room to DOWN
   • Takeover available here
9. Start room atomic take over on new connections status SYNCING
10. Sync with potential persistence
11. Room ready on new node: status UP

Notes

• The process should be independent for each room (we don't want non-persisted rooms to be held up on drains)
• We accept that there will be a xms pause in functionality for the sake of consistency
  • with yjs having a localdb provider, this will probably make this unobservable (?? not 100% on this)

Node State Flow

stateDiagram-v2
    [*] --> DOWN

    DOWN --> SYNCING: Start Sync

    state SYNCING {
        [*] --> SYNCING_LOAD

        SYNCING_LOAD --> SYNCING_SUCCESS: Load Success
        SYNCING_LOAD --> SYNCING_RETRY_LOAD: Load Fail

        SYNCING_RETRY_LOAD --> SYNCING_LOAD: Retry Load
        SYNCING_RETRY_LOAD --> SYNCING_FAIL: Retry Limit Exceeded

        SYNCING_SUCCESS --> [*]
    }

    SYNCING --> UP: Sync Complete

    UP --> DRAINING: Start Draining

    state DRAINING {
        [*] --> DRAINING_STORE

        DRAINING_STORE --> DRAINING_SUCCESS: Store Success
        DRAINING_STORE --> DRAINING_RETRY_STORE: Store Fail

        DRAINING_RETRY_STORE --> DRAINING_STORE: Retry Store
        DRAINING_RETRY_STORE --> DRAINING_FAIL: Retry Limit Exceeded


        DRAINING_SUCCESS --> [*]
    }

    DRAINING --> DOWN: Drain Complete

Loading

Changes Needed

• Room status UP, DOWN, DRAINING, SYNCING
• Relay takeover logic modification
• Persistence trait with noop default impl
• SIGINT trigger of drain
• Actual drain logic
• Update to TUI to include node status in table