fail to close db and still use the db after it's close

[july2993]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   see https://asktug.com/t/topic/33716/2
2. What did you expect to see?
   should no unexpect log like close the closed db and using the closed db.
3. What did you see instead?

  18 [2020/04/18 02:31:05.136 +00:00] [ERROR] [db.go:88] ["fail to close baseDB"] [task=mysql-replica-boe] [unit="binlog replication"] [error="[code=10004:class=database:scope=not-set:level=high] unexpect database error: close: sql: connection is already closed"]

7658 [2020/04/18 03:07:58.493 +00:00] [ERROR] [subtask.go:255] ["unit process error"] [subtask=mysql-replica-boe] [unit=Sync] ["error information"="{\"Type\":1,\"msg\":\"[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection      is already closed\\ngit.luolix.top/pingcap/dm/pkg/terror.(*Error).Delegate\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\\ngit.luolix.top/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\\n\\t/home/jenkins/a     gent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\\ngit.luolix.top/ping     cap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\\ngit.luolix.top/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\\n\\t/home/jenkins/agent/workspace/build_dm_master/g     o/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQL\\n\     \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).sync.func3\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\\ngit.luolix.top/pingcap     /dm/syncer.(*Syncer).sync\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).Run.func2\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/s     yncer.go:1093\\nruntime.goexit\\n\\t/usr/local/go/src/runtime/asm_amd64.s:1357\",\"error\":{\"ErrCode\":10006,\"ErrClass\":1,\"ErrLevel\":3,\"Message\":\"execute statement failed: begin: sql: connection is already closed\",\"RawCause\":\"sql: connection is already cl     osed\"}}"]
7659 [2020/04/18 03:07:58.493 +00:00] [ERROR] [subtask.go:255] ["unit process error"] [subtask=mysql-replica-boe] [unit=Sync] ["error information"="{\"Type\":1,\"msg\":\"[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection      is already closed\\ngit.luolix.top/pingcap/dm/pkg/terror.(*Error).Delegate\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\\ngit.luolix.top/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\\n\\t/home/jenkins/a     gent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\\ngit.luolix.top/ping     cap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\\ngit.luolix.top/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\\n\\t/home/jenkins/agent/workspace/build_dm_master/g     o/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQL\\n\     \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).sync.func3\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\\ngit.luolix.top/pingcap     /dm/syncer.(*Syncer).sync\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).Run.func2\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/s     yncer.go:1093\\nruntime.goexit\\n\\t/usr/local/go/src/runtime/asm_amd64.s:1357\",\"error\":{\"ErrCode\":10006,\"ErrClass\":1,\"ErrLevel\":3,\"Message\":\"execute statement failed: begin: sql: connection is already closed\",\"RawCause\":\"sql: connection is already cl     osed\"}}"]
7660 [2020/04/18 03:07:58.493 +00:00] [ERROR] [subtask.go:255] ["unit process error"] [subtask=mysql-replica-boe] [unit=Sync] ["error information"="{\"Type\":1,\"msg\":\"[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection      is already closed\\ngit.luolix.top/pingcap/dm/pkg/terror.(*Error).Delegate\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\\ngit.luolix.top/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\\n\\t/home/jenkins/a     gent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\\ngit.luolix.top/ping     cap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\\ngit.luolix.top/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\\n\\t/home/jenkins/agent/workspace/build_dm_master/g     o/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\\ngit.luolix.top/pingcap/dm/syncer.(*DBConn).executeSQL\\n\     \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).sync.func3\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\\ngit.luolix.top/pingcap     /dm/syncer.(*Syncer).sync\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).Run.func2\\n\\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/s     yncer.go:1093\\nruntime.goexit\\n\\t/usr/local/go/src/runtime/asm_amd64.s:1357\",\"error\":{\"ErrCode\":10006,\"ErrClass\":1,\"ErrLevel\":3,\"Message\":\"execute statement failed: begin: sql: connection is already closed\",\"RawCause\":\"sql: connection is already cl     osed\"}}"]

4. Versions of the cluster

   • DM version (run dmctl -V or dm-worker -V or dm-master -V):

     (paste DM version here, and you must ensure versions of dmctl, DM-worker and DM-master are same)
     

   • Upstream MySQL/MariaDB server version:

     (paste upstream MySQL/MariaDB server version here)
     

   • Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     (paste TiDB cluster version here)
     

   • How did you deploy DM: DM-Ansible or manually?

     (leave DM-Ansible or manually here)
     

   • Other interesting information (system version, hardware config, etc):

5. current status of DM cluster (execute query-status in dmctl)

6. Operation logs

   • Please upload dm-worker.log for every DM-worker instance if possible
   • Please upload dm-master.log if possible
   • Other interesting logs
   • Output of dmctl's commands with problems

7. Configuration of the cluster and the task

   • dm-worker.toml for every DM-worker instance if possible
   • dm-master.toml for DM-master if possible
   • task config, like task.yaml if possible
   • inventory.ini if deployed by DM-Ansible

8. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus for DM if possible

[csuzhangxc]

It should be fixed in #403.

[july2993]

It should be fixed in #403.