executor: avoid goroutine leak in TableReader along with memory leak

[fzhedu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

The source code only generates go routines, but not handle errors if some exceptions happens.
To to reproduce this issue using failpoint.

On the other hand, if the main thread is cancelled, those generated goroutines should stop correctly.

detail problems:

• TableReader->Next() is blocked, how to stop the execution once cancel the query manually?
• When call TableReader->Close(), how to stop coprocessor workers?
• When coprocessor workes panic, how to cancel the query correctly?
• what about IndexReader?

2. What did you expect to see? (Required)

all goroutine stops and return errors.

3. What did you see instead (Required)

other goroutine runs normally and occupy many memory .

4. Affected version (Required)

v4.0.x, v3

5. Root Cause Analysis

lack handing exceptions.

[g1eny0ung]

/label component/executor

[tiancaiamao]

Good questions, I'm glad to answer them:

1. TableReader->Next() is blocked, how to stop the execution once cancel the query manually?

Everytime executor.Next is called, the killed flag is checked.
If the code is blocked within TableReader->Next() and never run to the Next() function, we should check why it's blocked and check the flag there.

For example, we have already check the pessimistic lock waiting, each backoff and coprocessor finish one region.

2. When call TableReader->Close(), how to stop coprocessor workers?

The correct order should be coprocessor throw an error (maybe ErrInterrupted) to TableReader, then TableReader call Close() function.

We should not let TableReader peceive kill during the coprocessor worker is working, exception handling is hard that way.
Instead, only one place handle the kill flag and turn it into ErrInterrupted.
Exception handling is turned into error handling which is much easier.

3. When coprocessor workes panic, how to cancel the query correctly?

When the coprocessor workes panic, it should throw error to the caller -- TableReader, then the TableReader handle the error.

4. what about IndexReader?

The same with TableReader.

It's important to understand that there is no real cancel operation.
It's wrong to think like a caller preceive the killed flag and cancel the worker.
killed flag is checked only when some code runs to check it, in this way exception handling is turned to error handling.