-
Notifications
You must be signed in to change notification settings - Fork 412
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add DAGContext so InterpreterDAG can exchange information with DAGDriver #166
Conversation
} | ||
}; | ||
using FieldTpAndFlags = std::vector<FieldTpAndFlag>; | ||
class DAGContext |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Put an empty line between.
@@ -41,7 +28,7 @@ class DAGQuerySource : public IQuerySource | |||
static const String LIMIT_NAME; | |||
|
|||
DAGQuerySource(Context & context_, RegionID region_id_, UInt64 region_version_, UInt64 region_conf_version_, | |||
const tipb::DAGRequest & dag_request_); | |||
const tipb::DAGRequest & dag_request_, DAGContext & dag_context_); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd prefer put dag_context argument next to context, as a kind of 'context group'.
@@ -56,6 +43,12 @@ class DAGQuerySource : public IQuerySource | |||
bool hasTopN() const { return order_index != -1; }; | |||
bool hasLimit() const { return order_index == -1 && limit_index != -1; }; | |||
|
|||
Int32 getTSIndex() const { return ts_index; }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These should be of type size_t.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The initial val of these index is -1, so the type of the index is Int32
ASTPtr getAST() const { return ast; }; | ||
DAGContext & getDAGContext() const { return dag_context; }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Put it as above as possible, to reflect the position in the private segment.
{ | ||
public: | ||
DAGContext() = default; | ||
std::vector<BlockInputStreams> & getProfileStreamsList() { return profile_streams_list; }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the getter returns a non-const ref, I think we can directly make the member public.
@@ -30,7 +31,9 @@ void DAGDriver::execute() | |||
{ | |||
context.setSetting("read_tso", UInt64(dag_request.start_ts())); | |||
|
|||
DAGQuerySource dag(context, region_id, region_version, region_conf_version, dag_request); | |||
DAGContext dag_context; | |||
dag_context.getProfileStreamsList().resize(dag_request.executors_size()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can put this resize into dag context's ctor with a given size argument.
Since you'll use profile_streams_list[executor_index] to assign stream list to an executor id, not push_back()/emplace_back(), so it'll be meaningless to initialize an empty vector, which makes specifying the initial size of the vector required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
* basic framework for coprocessor support in tiflash * basic support for InterpreterDagRequestV2 * code refine * tipb submodule use tipb master branch * rewrite build flow in InterpreterDagRequest * rename Dag to DAG * Update tipb submodule * basic support for selection/limit/topn executor in InterpreterDAGRequest * basic support for selection/limit/topn executor in InterpreterDAGRequest (#150) * merge pingcap/cop branch * Code reorg * Format * merge pingcap/cop * Refine code * basic support for dag agg executor * Code refine * Refine code * Another way of getting codec flag * fix cop test regression (#157) * fix cop test regression * address comments * format code * fix npe during dag execute (#160) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * Add tipb cpp gen in build script * Fix build error and adjust some formats * Fix build error * Fix build error * Update flash configs * Format * throw exception when meet error duing cop request handling (#162) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver (#166) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema (#167) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema * Move flash/cop/dag to individual library * DAG planner fix and mock dag request (#169) * Enhance dbg invoke and add dag as schemaful function * Add basic sql parse to dag * Column id starts from 1 * Fix value to ref * Add basic dag test * Fix dag bugs and pass 1st mock test * Make dag go normal routine and add mock dag * Add todo * Add comment * Fix gcc compile error * Enhance dag test * Address comments * Fix DAG get and lock storage * handle error in cop request (#171) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema * handle error in coprocessor request * refine code * use Clear to clear a protobuf message completely * refine code * code refine && several minor bug fix (#174) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema * handle error in coprocessor request * refine code * use Clear to clear a protobuf message completely * refine code * code refine && several minor bug fix * address comments * address comments * Fix region id in mock dag * support udf in (#175) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema * handle error in coprocessor request * refine code * use Clear to clear a protobuf message completely * refine code * code refine && several minor bug fix * address comments * address comments * support udf in * refine code * address comments * address comments * 1. fix decode literal expr error, 2. add all scalar function sig in scalar_func_map (#177) * add all scalar function sig in scalarFunMap * fix literal expr decode * enable ltrim && rtrim * code refine * use throw instead of rethrow in DAGDriver.cpp * some bug fix (#179) * add all scalar function sig in scalarFunMap * fix literal expr decode * enable ltrim && rtrim * code refine * use throw instead of rethrow in DAGDriver.cpp * 1. fix decode UInt literal error, 2. support mysqlDecimal type * format code * Support all DAG operator types in mock SQL -> DAG parser (#176) * Enhance dbg invoke and add dag as schemaful function * Add basic sql parse to dag * Column id starts from 1 * Fix value to ref * Add basic dag test * Fix dag bugs and pass 1st mock test * Make dag go normal routine and add mock dag * Add todo * Add comment * Fix gcc compile error * Enhance dag test * Address comments * Enhance mock sql -> dag compiler and add project test * Mock sql dag compiler support more expression types and add filter test * Add topn and limit test * Add agg for sql -> dag parser and agg test * Add dag specific codec * type * Update codec accordingly * Remove cop-test * filter column must be uint8 in tiflash (#180) * add all scalar function sig in scalarFunMap * fix literal expr decode * enable ltrim && rtrim * code refine * use throw instead of rethrow in DAGDriver.cpp * 1. fix decode UInt literal error, 2. support mysqlDecimal type * format code * filter column must be uint8 in tiflash * address comments * address comments * address comments * remove useless include * 1. fix encode null error, 2. fix empty field type generated by TiFlash (#182) * add all scalar function sig in scalarFunMap * fix literal expr decode * enable ltrim && rtrim * code refine * use throw instead of rethrow in DAGDriver.cpp * 1. fix decode UInt literal error, 2. support mysqlDecimal type * format code * filter column must be uint8 in tiflash * address comments * address comments * address comments * remove useless include * 1. fix encode null error, 2. fix empty field type generated by TiFlash * check validation of dag exprs field type (#183) * check validation of dag exprs field type * format code * address comments * add more coprocessor mock tests (#185) * check validation of dag exprs field type * format code * address comments * add more filter test * add data type tests * remove useless comment * disable decimal test * add some log about implicit cast (#188) * check validation of dag exprs field type * format code * address comments * add more filter test * add data type tests * remove useless comment * disable decimal test * add some log about implicit cast * address comment * Pass DAG tests after merging master (#199) * Enhance dbg invoke and add dag as schemaful function * Add basic sql parse to dag * Column id starts from 1 * Fix value to ref * Add basic dag test * Fix dag bugs and pass 1st mock test * Make dag go normal routine and add mock dag * Add todo * Add comment * Fix gcc compile error * Enhance dag test * Address comments * Enhance mock sql -> dag compiler and add project test * Mock sql dag compiler support more expression types and add filter test * Add topn and limit test * Add agg for sql -> dag parser and agg test * Add dag specific codec * type * Update codec accordingly * Remove cop-test * Pass tests after merging master * Fix date/datetime/bit encode error (#200) * Enhance dbg invoke and add dag as schemaful function * Add basic sql parse to dag * Column id starts from 1 * Fix value to ref * Add basic dag test * Fix dag bugs and pass 1st mock test * Make dag go normal routine and add mock dag * Add todo * Add comment * Fix gcc compile error * Enhance dag test * Address comments * Enhance mock sql -> dag compiler and add project test * Mock sql dag compiler support more expression types and add filter test * Add topn and limit test * Add agg for sql -> dag parser and agg test * Add dag specific codec * type * Update codec accordingly * Remove cop-test * Pass tests after merging master * Copy some changes from xufei * Enable date/datetime test * Enable date/datetime test * Refine code * Adjust date/datetime tiflash rep to UInt * Fix datetime to Int * Typo * improve dag execution time collection (#202) * improve dag execution time collection * address comment * update comments * update comment * update comment * column id in table scan operator may be -1 (#205) * improve dag execution time collection * address comment * update comments * update comment * update comment * column id in table scan operator may be -1 * column id in table scan operator may be -1 * quick fix for decimal encode (#210) * quick fix for decimal encode * address comments * update comments * support udf like with 3 arguments (#212) * support udf like with 3 arguments * address comments * add some comments * Flash-473 optimize date and datetime comparison (#221) * support udf like with 3 arguments * address comments * add some comments * Flash-473 optimize date and datetime comparison * address comments * FLASH-479 select from empty table throw error in tiflash (#223) * 1. select from empty table throw error in tiflash, 2. add some logs, 3. disable timestamp literal in DAG request * revert unrelated change * Update flash service port * fix bug in DAGBlockOutputStream (#230) * FLASH-475: Support BATCH COMMANDS in flash service (#232) * Initial batch command support * Add config to control thread pool size * Address comments * FLASH-483: Combine raft service and flash service (#235) * Combine raft service and flash service * Address comment and fix build error * Update configs * Fix build error * Fix test regression * Fix null value bug in datum * FLASH-490: Fix table scan with -1 column ID and no agg (#240) * Fix table scan with -1 column ID and no agg * Add break * Remove useless includes * Use dag context to store void ft instead of dag query source * Fix decimal type reverse get * Change adding smallest column to adding handle column, address comments * throw error if the cop request is not based on full region scan (#247) * throw error if the cop request is not based on full region scan * format code * FLASH-437 Support time zone in coprocessor (#259) * do not allow timestamp literal in DAG request * refine code * fix cop date type encode error * support tz info in DAG request * address comments * Address comment * FLASH-489 support key condition for coprocessor query (#261) * support key condition for coprocessor query * add tests * remove useless code * check validation when build RPNElement for function in/notIn * address comments * address comments * only return execute summaies if requested (#264) * Refine service init (#265) * FLASH-554 cop check range should be based on region range (#270) * only return execute summaies if requested * cop check range should be based on region range * address comments * add tests * minor improve * minor improve (#273) * Fix mutex on timezone retrieval (#276) * fix mutex contention * add const ref * Fix race condition of batch command handling (#277) * address comment * address comments * address comments * Fix NULL order for dag (#281) * refine get actions in DAGExpressionAnalyzer, fix bug in dbgFuncCoprocessor (#282) * remove duplicate agg funcs (#283) * 1. remove duplicate agg funcs, 2. for column ref expr, change column_id to column_index since the value stored in column ref expr is not column id * bug fix * address comments * Update dbms/src/Flash/BatchCommandsHandler.cpp Co-Authored-By: JaySon <jayson.hjs@gmail.com> * revert unnecessary changes
* basic framework for coprocessor support in tiflash * basic support for InterpreterDagRequestV2 * code refine * tipb submodule use tipb master branch * rewrite build flow in InterpreterDagRequest * rename Dag to DAG * Update tipb submodule * basic support for selection/limit/topn executor in InterpreterDAGRequest * basic support for selection/limit/topn executor in InterpreterDAGRequest (#150) * merge pingcap/cop branch * Code reorg * Format * merge pingcap/cop * Refine code * basic support for dag agg executor * Code refine * Refine code * Another way of getting codec flag * fix cop test regression (#157) * fix cop test regression * address comments * format code * fix npe during dag execute (#160) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * Add tipb cpp gen in build script * Fix build error and adjust some formats * Fix build error * Fix build error * Update flash configs * Format * throw exception when meet error duing cop request handling (#162) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver (#166) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema (#167) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema * Move flash/cop/dag to individual library * DAG planner fix and mock dag request (#169) * Enhance dbg invoke and add dag as schemaful function * Add basic sql parse to dag * Column id starts from 1 * Fix value to ref * Add basic dag test * Fix dag bugs and pass 1st mock test * Make dag go normal routine and add mock dag * Add todo * Add comment * Fix gcc compile error * Enhance dag test * Address comments * Fix DAG get and lock storage * handle error in cop request (#171) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema * handle error in coprocessor request * refine code * use Clear to clear a protobuf message completely * refine code * code refine && several minor bug fix (#174) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema * handle error in coprocessor request * refine code * use Clear to clear a protobuf message completely * refine code * code refine && several minor bug fix * address comments * address comments * Fix region id in mock dag * support udf in (#175) * fix cop test regression * address comments * format code * fix npe for dag execute * format code * address comment * add some comments * throw exception when meet error duing cop request handling * address comments * add error code * throw exception when meet error duing cop request handling * address comments * add DAGContext so InterpreterDAG can exchange information with DAGDriver * fix bug * 1. refine code, 2. address comments * update comments * columnref index is based on executor output schema * handle error in coprocessor request * refine code * use Clear to clear a protobuf message completely * refine code * code refine && several minor bug fix * address comments * address comments * support udf in * refine code * address comments * address comments * 1. fix decode literal expr error, 2. add all scalar function sig in scalar_func_map (#177) * add all scalar function sig in scalarFunMap * fix literal expr decode * enable ltrim && rtrim * code refine * use throw instead of rethrow in DAGDriver.cpp * some bug fix (#179) * add all scalar function sig in scalarFunMap * fix literal expr decode * enable ltrim && rtrim * code refine * use throw instead of rethrow in DAGDriver.cpp * 1. fix decode UInt literal error, 2. support mysqlDecimal type * format code * Support all DAG operator types in mock SQL -> DAG parser (#176) * Enhance dbg invoke and add dag as schemaful function * Add basic sql parse to dag * Column id starts from 1 * Fix value to ref * Add basic dag test * Fix dag bugs and pass 1st mock test * Make dag go normal routine and add mock dag * Add todo * Add comment * Fix gcc compile error * Enhance dag test * Address comments * Enhance mock sql -> dag compiler and add project test * Mock sql dag compiler support more expression types and add filter test * Add topn and limit test * Add agg for sql -> dag parser and agg test * Add dag specific codec * type * Update codec accordingly * Remove cop-test * filter column must be uint8 in tiflash (#180) * add all scalar function sig in scalarFunMap * fix literal expr decode * enable ltrim && rtrim * code refine * use throw instead of rethrow in DAGDriver.cpp * 1. fix decode UInt literal error, 2. support mysqlDecimal type * format code * filter column must be uint8 in tiflash * address comments * address comments * address comments * remove useless include * 1. fix encode null error, 2. fix empty field type generated by TiFlash (#182) * add all scalar function sig in scalarFunMap * fix literal expr decode * enable ltrim && rtrim * code refine * use throw instead of rethrow in DAGDriver.cpp * 1. fix decode UInt literal error, 2. support mysqlDecimal type * format code * filter column must be uint8 in tiflash * address comments * address comments * address comments * remove useless include * 1. fix encode null error, 2. fix empty field type generated by TiFlash * check validation of dag exprs field type (#183) * check validation of dag exprs field type * format code * address comments * add more coprocessor mock tests (#185) * check validation of dag exprs field type * format code * address comments * add more filter test * add data type tests * remove useless comment * disable decimal test * add some log about implicit cast (#188) * check validation of dag exprs field type * format code * address comments * add more filter test * add data type tests * remove useless comment * disable decimal test * add some log about implicit cast * address comment * Pass DAG tests after merging master (#199) * Enhance dbg invoke and add dag as schemaful function * Add basic sql parse to dag * Column id starts from 1 * Fix value to ref * Add basic dag test * Fix dag bugs and pass 1st mock test * Make dag go normal routine and add mock dag * Add todo * Add comment * Fix gcc compile error * Enhance dag test * Address comments * Enhance mock sql -> dag compiler and add project test * Mock sql dag compiler support more expression types and add filter test * Add topn and limit test * Add agg for sql -> dag parser and agg test * Add dag specific codec * type * Update codec accordingly * Remove cop-test * Pass tests after merging master * Fix date/datetime/bit encode error (#200) * Enhance dbg invoke and add dag as schemaful function * Add basic sql parse to dag * Column id starts from 1 * Fix value to ref * Add basic dag test * Fix dag bugs and pass 1st mock test * Make dag go normal routine and add mock dag * Add todo * Add comment * Fix gcc compile error * Enhance dag test * Address comments * Enhance mock sql -> dag compiler and add project test * Mock sql dag compiler support more expression types and add filter test * Add topn and limit test * Add agg for sql -> dag parser and agg test * Add dag specific codec * type * Update codec accordingly * Remove cop-test * Pass tests after merging master * Copy some changes from xufei * Enable date/datetime test * Enable date/datetime test * Refine code * Adjust date/datetime tiflash rep to UInt * Fix datetime to Int * Typo * improve dag execution time collection (#202) * improve dag execution time collection * address comment * update comments * update comment * update comment * column id in table scan operator may be -1 (#205) * improve dag execution time collection * address comment * update comments * update comment * update comment * column id in table scan operator may be -1 * column id in table scan operator may be -1 * quick fix for decimal encode (#210) * quick fix for decimal encode * address comments * update comments * support udf like with 3 arguments (#212) * support udf like with 3 arguments * address comments * add some comments * Flash-473 optimize date and datetime comparison (#221) * support udf like with 3 arguments * address comments * add some comments * Flash-473 optimize date and datetime comparison * address comments * FLASH-479 select from empty table throw error in tiflash (#223) * 1. select from empty table throw error in tiflash, 2. add some logs, 3. disable timestamp literal in DAG request * revert unrelated change * Update flash service port * fix bug in DAGBlockOutputStream * fix bug in DAGBlockOutputStream (#230) * FLASH-475: Support BATCH COMMANDS in flash service (#232) * Initial batch command support * Add config to control thread pool size * Address comments * init change for array encode * FLASH-483: Combine raft service and flash service (#235) * Combine raft service and flash service * Address comment and fix build error * Update configs * Fix build error * Fix test regression * Fix null value bug in datum * FLASH-490: Fix table scan with -1 column ID and no agg (#240) * Fix table scan with -1 column ID and no agg * Add break * Remove useless includes * Use dag context to store void ft instead of dag query source * Fix decimal type reverse get * Change adding smallest column to adding handle column, address comments * throw error if the cop request is not based on full region scan (#247) * throw error if the cop request is not based on full region scan * format code * FLASH-437 Support time zone in coprocessor (#259) * do not allow timestamp literal in DAG request * refine code * fix cop date type encode error * support tz info in DAG request * address comments * Address comment * use the new date implementation * FLASH-489 support key condition for coprocessor query (#261) * support key condition for coprocessor query * add tests * remove useless code * check validation when build RPNElement for function in/notIn * address comments * address comments * only return execute summaies if requested (#264) * refine code * Refine service init (#265) * fix bug * fix bug * FLASH-554 cop check range should be based on region range (#270) * only return execute summaies if requested * cop check range should be based on region range * address comments * add tests * minor improve * add ut for arrow encode * minor improve (#273) * update tipb * Fix mutex on timezone retrieval (#276) * fix mutex contention * add const ref * Fix race condition of batch command handling (#277) * update tipb version * set default record_per_chunk to 1024 * address comment * address comments * refine code * refine code * add mock_dag test * code refine * code refine * address comments * Fix NULL order for dag (#281) * refine get actions in DAGExpressionAnalyzer, fix bug in dbgFuncCoprocessor (#282) * remove duplicate agg funcs (#283) * 1. remove duplicate agg funcs, 2. for column ref expr, change column_id to column_index since the value stored in column ref expr is not column id * bug fix * refine code * remove useless code * address comments * remove uselss include * address comments * refine code * address comments * format code * fix typo * Update dbms/src/Flash/BatchCommandsHandler.cpp Co-Authored-By: JaySon <jayson.hjs@gmail.com> * revert unnecessary changes * refine code * fix build error * refine code * address comments * refine code * address comments
…#166) Signed-off-by: Wish <breezewish@outlook.com>
…pingcap#166) Signed-off-by: Wish <breezewish@outlook.com>
…pingcap#166) Signed-off-by: Wish <breezewish@outlook.com>
DAGContext
soInterpreterDAG
can exchange information withDAGDriver
DAGExpressionAnalyzer::appendAggSelect