DM failed to resume-task after ansible-playbook rolling_update.yml --tags=tidb.

[jianhaiqing]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

   • ansible-playbook rolling_update.yml --tags=tidb ( from 2.1.2 to 2.1.3 )
   • ./dmctl/dmctl --master-addr 10.111.210.242:8261
   • query-status ( task paused, expected )
   • resume-task -w "10.111.21.216:8262" seewo (query-status, still paused )
   • resume-task -w "10.111.21.216:8262" seewo (query-status, still paused )
   • resume-task -w "10.111.21.216:8262" seewo ( successfully )

2. What did you expect to see?

   • firstly resum-task should successfully continue.

3. What did you see instead?

• first resume-task

"result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "invalid connection\ngit.luolix.top/pingcap/errors.AddStack\n\t/home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/errors@v0.11.0/errors.go:174\ngit.luolix.top/pingcap/errors.Trace\n\t/home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/errors@v0.11.0/juju_adaptor.go:12\ngit.luolix.top/pingcap/dm/syncer.(*Conn).querySQL\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:93\ngit.luolix.top/pingcap/dm/syncer.getTableColumns\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:351\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).getTableFromDB\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:511\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).getTable\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:537\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).Run\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:1137\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).Process\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:446\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).Resume\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:1916\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"
                            }
                        ],
                        "detail": null
                    }

+ second resume-task

"result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "ExecSQL",
                                "msg": "invalid connection\ngit.luolix.top/pingcap/errors.AddStack\n\t/home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/errors@v0.11.0/errors.go:174\ngit.luolix.top/pingcap/errors.Trace\n\t/home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/errors@v0.11.0/juju_adaptor.go:12\ngit.luolix.top/pingcap/dm/syncer.(*Conn).executeSQLJobImp\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:222\ngit.luolix.top/pingcap/dm/syncer.(*Conn).executeSQLJob\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:195\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:733\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:806\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:867\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "invalid connection\ngit.luolix.top/pingcap/errors.AddStack\n\t/home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/errors@v0.11.0/errors.go:174\ngit.luolix.top/pingcap/errors.Trace\n\t/home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/errors@v0.11.0/juju_adaptor.go:12\ngit.luolix.top/pingcap/dm/syncer.(*Conn).executeSQLJobImp\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:222\ngit.luolix.top/pingcap/dm/syncer.(*Conn).executeSQLJob\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:195\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:733\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:806\ngit.luolix.top/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:867\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"
                            },{}, --- { total: 13 Type : ExecSQL}

4. Versions of the cluster

   • DM version (run dmctl -V or dm-worker -V or dm-master -V):

     ./dmctl/dmctl -V
     

Release Version: v1.0.0-alpha-10-g4d01d79
Git Commit Hash: 4d01d79
Git Branch: master
UTC Build Time: 2019-02-11 14:50:57
Go Version: go version go1.11.2 linux/amd64
```

- Upstream MySQL server version:

    ```
   Server version:		5.7.24-27-log Source distribution
    ```

- Downstream TiDB cluster version (execute `SELECT tidb_version();` in a MySQL client):

    ```
   2.1.3
    ```

- How did you deploy DM: DM-Ansible or manually?

    ```
     DM-Ansible
    ```

- Other interesting information (system version, hardware config, etc):

    >

[csuzhangxc]

this should be already fixed in #403. (in DM v1.0.4).