support system variable wait_timeout. (#8245)

[hhu-cc]

What problem does this PR solve?

close client connection when the server waited for a long time , support system variable wait_timeout to define the wait time. #8245

What is changed and how it works?

use TCPConn.SetReadDeadline() before readPacket()

Check List

Tests

• Integeration test

Code changes

• Has exported variable/fields change

Related changes

• Need to be included in the release note

---

This change is 

[sre-bot]

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

[shenli]

@hhxcc Thanks! Please add a test case.

[shenli]

/ok-to-test

[shenli]

/run-all-tests

[hhu-cc]

@shenli I can only see a litter test case in conn_test.go, could you give some advise to write test case ?

[shenli]

@lysu Please help @hhxcc to add a test case.

[hhu-cc]

@lysu PTAL . I have do some refactor, add one test case , bu still do not know how to test the function readPacketWithTimeout.

[lysu]

@hhxcc I will try to do it in tomorrow tks~

[lysu]

/run-all-tests tidb-test=pr/654

[lysu]

func TestConnectionWaitTimeout(t *testing.T) {
 	// this test does not use sql.database, it uses the driver directly
 	if !available {
 		t.Skipf("MySQL server not running on %s", netAddr)
 	}

 	idleDB, err := sql.Open("mysql", dsn)
 	if err != nil {
 		t.Fatalf("error connecting: %s", err.Error())
 	}
 	defer idleDB.Close()
 	_, err = idleDB.Exec("create database if not exists waittimeouttest")
 	assertNoError(t, err)
 	_, err = idleDB.Exec("use waittimeouttest")
 	assertNoError(t, err)

 	checkDB, err := sql.Open("mysql", dsn)
 	if err != nil {
 		t.Fatalf("error connecting: %s", err.Error())
 	}
 	defer checkDB.Close()
 	r := checkDB.QueryRow("select count(1) from information_schema.processlist where db = 'waittimeouttest'")
 	var pc int
 	err = r.Scan(&pc)
 	assertNoError(t, err)
 	assertEqual(t, pc, 1)

 	_, err = idleDB.Exec("set @@wait_timeout=1")
 	assertNoError(t, err)

 	time.Sleep(2*time.Second)

 	r = checkDB.QueryRow("select count(1) from information_schema.processlist where db = 'waittimeouttest'")
 	err = r.Scan(&pc)
 	assertNoError(t, err)
 	assertEqual(t, pc, 0)
 }

@hhxcc I just add a simple testcase in integeration repo which will setup a tidbserver and test them via network.

[hhu-cc]

@lysu thanks a lot.

[lysu]

It's better use this way to detect timeout error

		start := time.Now()
		data, err := cc.readPacketWithTimeout()
		idleTime := time.Now().Sub(start)
		if err != nil {
			if terror.ErrorNotEqual(err, io.EOF) {
				if netErr, isNetErr :=  errors.Cause(err).(net.Error); isNetErr && netErr.Timeout() {
					log.Infof("con:%d read packet timeout, close this connection, idle: %v, timeout: %v", cc.connectionID, idleTime, cc.getSessionVarsWaitTimeout())
				} else {
					errStack := errors.ErrorStack(err)
					if !strings.Contains(errStack, "use of closed network connection") {
						log.Errorf("con:%d read packet error, close this connection %s",
							cc.connectionID, errStack)
					}
				}
			}
			return
		}

detect close maybe be improve in furture too - - golang/go#4373

[lysu]

@hhxcc here, maybe we also take care about a not easy question:

now, deadline is set on readPacket, but one readPacket maybe trigger several readOnePacket then go though the buffer and trigger or not more read() call on wire and deadline works on final read.

so for a corner case: someone make a big data input to tidb-server, then mysql protocol will transfer them in multiple packet, first packet arrive time is t1, last packet arrive time is t2, and t2 - t1 > wait_time, current PR will give a error.

but I think wait_timeout should be "the timeout that no data on the wire", we should close connection only when it's idle, no matter how, read deadline should be set for per unbuffed read(), just like https://github.com/go-sql-driver/mysql/blob/6be42e0ff99645d7d9626d779001a46e39c5f280/buffer.go#L63 does and it seems need change buffer- -?

at last, we should take care situation that one conn read with deadline and later read without deadline, once deadline be set, later read need be reset new deadline or 0(no timeout) or it will got unexpect timeout error.

[hhu-cc]

@lysu thank you for the explanation with so much details, I will add SetReadDeadline before each readOnePacket .
If there are any other problem , please let me know.
thanks again.

[lysu]

@hhxcc I think set on each readOnePacket fix mainly question, although set on ech read is more better, I'm not confirm whelther bufferedConn will introduce other question in some corner case~?

@tiancaiamao @jackysp how do you think about this?

[hhu-cc]

@lysu PTAL SetReadDeadline has been added before each io.ReadFull , and change the way to detect timeout error.

[lysu]

/run-all-tests tidb-test=pr/654

[lysu]

LGTM

&

@jackysp PTAL thx

[jackysp]

It's better to define a const for 28800.

[hhu-cc]

fixed

[lysu]

/run-sqllogic-test

[jackysp]

Thanks for your quick fix, @hhxcc ! But we usually define the default values of system variable at

tidb/sessionctx/variable/tidb_vars.go

Line 221 in e69aa27

const (

[hhu-cc]

ok , it has been moved to tidb_vars.go

[jackysp]

LGTM