feat(cubestore): Initial queue support

[ovr]

Hello!

refs #5042

General queue:

sequenceDiagram
    participant BackgroundQueryQueue
    participant QueueQueue
    participant QueueDriverInterface
    participant CubeStore

    QueueQueue->>QueueDriverInterface: getResult
    QueueDriverInterface->>+CubeStore: QUEUE RESULT ?
    QueueDriverInterface-->>+QueueQueue: QueryResult|null
    deactivate CubeStore

    QueueQueue->>QueueDriverInterface: addToQueue
    QueueDriverInterface->>+CubeStore: QUEUE ADD PRIORITY N key payload
    QueueDriverInterface-->>+QueueQueue: AddToQueueResponse

    loop reconcileQueueImpl
        QueueQueue->>QueueDriverInterface: getQueriesToCancel
        QueueQueue->>QueueDriverInterface: getQueryAndRemove
        QueueDriverInterface->>CubeStore: QUEUE CANCEL

        QueueQueue->>QueueDriverInterface: getActiveQueries
        QueueDriverInterface->>CubeStore: QUEUE TODO
        QueueDriverInterface-->>+QueueQueue: getActiveQueriesResponse

        QueueQueue->>QueueDriverInterface: getToProcessQueries
        QueueDriverInterface->>CubeStore: QUEUE TODO
        QueueDriverInterface-->>+QueueQueue: getToProcessQueriesResponse

        QueueQueue-)+BackgroundQueryQueue: processQuery
        Note over QueueQueue,BackgroundQueryQueue: Async call to procesQuery, which doesnt block here
    end

    alt lookUpInActive: Lookup query in processing
        QueueQueue->>QueueDriverInterface: getQueryDef
        activate CubeStore
        QueueDriverInterface->>CubeStore: QUEUE GET ?key
        CubeStore-->>+QueueQueue: QueryDef|null
        deactivate CubeStore

        QueueQueue->>QueueDriverInterface: getQueryStageState
        activate CubeStore
        QueueDriverInterface->>CubeStore: TODO
        CubeStore-->>+QueueQueue: TODO
        deactivate CubeStore
        Note over QueueQueue,QueueDriverInterface: Show waiting for query
    end

    QueueQueue->>QueueDriverInterface: getResultBlocking
    activate CubeStore
    QueueDriverInterface->>CubeStore: QUEUE RESULT_BLOCKING ?timeout ?key
    CubeStore-->>+QueueQueue: QueryResult|null
    deactivate CubeStore

Loading

Background execution process:

sequenceDiagram
    participant QueryOrchestrator
    participant BackgroundQueryQueue
    participant QueueDriverInterface
    participant CubeStore

    loop processQuery: Background execution
        BackgroundQueryQueue->>QueueDriverInterface: getNextProcessingId
        activate CubeStore
        QueueDriverInterface->>CubeStore: CACHE INCR ?
        CubeStore-->>+BackgroundQueryQueue: number
        deactivate CubeStore

        BackgroundQueryQueue->>QueueDriverInterface: retrieveForProcessing
        activate CubeStore
        QueueDriverInterface->>CubeStore: QUEUE RETRIEVE CONCURRENCY ?number ?key
        CubeStore-->>+BackgroundQueryQueue: QueryDef
        deactivate CubeStore

        BackgroundQueryQueue->>QueueDriverInterface: optimisticQueryUpdate
        activate CubeStore
        QueueDriverInterface->>CubeStore: QUEUE MERGE_EXTRA ?key {"startTime"}
        CubeStore-->>+BackgroundQueryQueue: ok
        deactivate CubeStore

        BackgroundQueryQueue->>QueueDriverInterface: optimisticQueryUpdate
        activate CubeStore
        QueueDriverInterface->>CubeStore: QUEUE MERGE_EXTRA ?key {"cancelHandler"}
        CubeStore-->>+BackgroundQueryQueue: ok
        deactivate CubeStore

        par executing: Query
            BackgroundQueryQueue->>QueueDriverInterface: updateHeartBeat
            QueueDriverInterface-->>BackgroundQueryQueue: ok
            Note over BackgroundQueryQueue,QueueDriverInterface: intervalTimer

            BackgroundQueryQueue->>QueryOrchestrator: execute
            QueryOrchestrator-->>BackgroundQueryQueue: result
        end

        BackgroundQueryQueue->>QueueDriverInterface: setResultAndRemoveQuery
        activate CubeStore
        QueueDriverInterface->>CubeStore: QUEUEU ACK ?key ?result
        CubeStore-->>+BackgroundQueryQueue: ok
        deactivate CubeStore
    end

Loading

Thanks

[codecov]

Codecov Report

Base: 41.33% // Head: 41.33% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (1a61710) compared to base (235ec70).
Patch coverage: 60.93% of modified lines in pull request are covered.

❗ Current head 1a61710 differs from pull request most recent head f3811ae. Consider uploading reports for the commit f3811ae to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #5541   +/-   ##
=======================================
  Coverage   41.33%   41.33%           
=======================================
  Files         151      151           
  Lines       20014    20027   +13     
  Branches     5206     5208    +2     
=======================================
+ Hits         8272     8278    +6     
- Misses      10904    10910    +6     
- Partials      838      839    +1     

Flag	Coverage Δ
cube-backend	41.33% <60.93%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...y-orchestrator/src/orchestrator/PreAggregations.ts	78.53% <ø> (ø)
...orchestrator/src/orchestrator/QueryOrchestrator.ts	55.00% <ø> (ø)
...-orchestrator/src/orchestrator/RedisQueueDriver.js	2.59% <0.00%> (-0.09%)	⬇️
...ges/cubejs-server-core/src/core/OrchestratorApi.ts	68.80% <ø> (ø)
...-query-orchestrator/src/orchestrator/QueryQueue.js	62.72% <64.70%> (+0.34%)	⬆️
...-orchestrator/src/orchestrator/LocalQueueDriver.js	90.16% <71.42%> (-0.85%)	⬇️
...y-orchestrator/src/orchestrator/BaseQueueDriver.ts	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ovr]

🕺

[waralexrom]

Rust part LGTM 👍