Tracking issue for dataflow framework

[discord9]

What problem does the new feature solve?

Being able to do simple continuous aggregation.

What does the feature do?

• not a complete streaming-processing system. Only a must subset functionalities are provided.
• can handle most aggregate operators within one table(i.e. Sum, avg, min, max and comparison operators). But others (join, trigger, txn etc.) are not the target feature.
  Framework

Implementation challenges

• Persisent intermediate state for operator
• Write more operator suitable for stream computation
• Limit the scope of system boundary to simple aggregation.

Implementation Progress

• RFC docs: RFC of Dataflow Framework #3185
• Basic Types, Expressions&Functions:
  • feat: basic types for Dataflow Framework #3186
  • feat: Basic Definitions for Expression&Functions for Dataflow #3267
  • feat(flow): impl ScalarExpr&Scalar Function #3283
  • feat(flow): impl for MapFilterProject #3359
  • feat(flow): accumlator for aggr func #3396
• Dataflow Logical Plan Defination & Execution
  • feat(flow): plan def #3490
  • feat(flow): shared in-memory state for dataflow operator  #3508 and fix(flow): Arrange get range with batch unaligned #3552
  • feat(flow): render map&related tests #3581
  • feat(flow): flow runtime render reduce operator #3736
• SQL/substrait to Dataflow Logical Plan translator
  • refactor(flow): func spec api&use Error not EvalError in mfp #3657
  • feat(flow): transform substrait SELECT&WHERE&GROUP BY to Flow Plan #3690
• register tasks to metasrc, and coordinate everything to make flow useable Tracking issue for intercorporation between Flow and Metasrv #3664

[tisonkun]

@discord9 I read the RFC now and wonder what's a completed sample for this feature.

I can see how to create a task (continuous query/materialize view):

CREATE TASK avg_over_5m WINDOW_SIZE = "5m" AS SELECT avg(value) FROM table WHERE time > now() - 5m GROUP BY time(1m)

Then we can use avg_over_5m as a normal table reference in query?

[discord9]

@discord9 I read the RFC now and wonder what's a completed sample for this feature.

I can see how to create a task (continuous query/materialize view):

CREATE TASK avg_over_5m WINDOW_SIZE = "5m" AS SELECT avg(value) FROM table WHERE time > now() - 5m GROUP BY time(1m)

Then we can use avg_over_5m as a normal table reference in query?

Yes, sorry for the late reply, github's layout for issues is really terrible, this task also create a result table avg_over_5m and write to it with negligible delay, so naturally one can use avg_over_5m in normal query

[killme2008]

@discord9 I think we can close this issue right now. The next iteration could start with a new issue. What do you think?

[discord9]

Close this issue as now have a basic dataflow framework, and can start a new issue to track it's next iteration