tests: refactor fault trigger #896

xiaojingchen · 2019-09-10T06:34:57Z

What problem does this PR solve?

support qm as vm manager
simplified the fault trigger

What is changed and how does it work?

Add a new VM manager to support qm.
Change the stability config, add the map of VM name and VM IP, avoid fault-trigger agent dynamic acquisition of IP

new stability config file:

   block_writer:
      concurrency: 12
    nodes:
      - physical_node: 172.16.5.11
        nodes:
        - ip: 172.16.4.247
          name: 105
      - physical_node: 172.16.5.26
        nodes:
        - ip: 172.16.4.133
          name: 200
      - physical_node: 172.16.5.27
        nodes:
        - ip: 172.16.4.121
          name: 203
      - physical_node: 172.16.5.28
        nodes:
        - ip: 172.16.4.139
          name: 204
      - physical_node: 172.16.5.29
        nodes:
        - ip: 172.16.5.147
          name: 137
        - ip: 172.16.5.148
          name: 138
    etcds:
      - physical_node: 172.16.5.11
        nodes:
        - ip: 172.16.4.247
          name: 105
      - physical_node: 172.16.5.26
        nodes:
        - ip: 172.16.4.133
          name: 200
      - physical_node: 172.16.5.27
        nodes:
        - ip: 172.16.4.121
          name: 203
    apiservers:
      - physical_node: 172.16.5.11
        nodes:
        - ip: 172.16.4.247
          name: 105
      - physical_node: 172.16.5.26
        nodes:
        - ip: 172.16.4.133
          name: 200
      - physical_node: 172.16.5.27
        nodes:
        - ip: 172.16.4.121
          name: 203

Check List

Tests

Unit test
E2E test
Stability test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has Helm charts change
Has Go code change

Side effects

Breaking backward compatibility

Related changes

NONE

Does this PR introduce a user-facing change?:

NONE

cofyc · 2019-09-10T06:44:08Z

tests/pkg/fault-trigger/manager/vm.go

+	StartVM(*VM) error
+}
+
+type VirshVMManager struct {


add a new file for VirshVMManager, e.g. vm_virsh.go and a new file for VirshVMManager, e.g. vm_qm.go what do you think?

weekface · 2019-09-10T08:43:42Z

tests/pkg/fault-trigger/manager/vm.go

@@ -225,3 +129,81 @@ func stripEmpty(data string) string {
 	}
 	return strings.Join(stripLines, "\n")
 }
+
+type QMVMManager struct {
+	*sync.RWMutex


this RWMutex is not used.

xiaojingchen · 2019-09-11T08:45:21Z

@weekface @cofyc @onlymellb @tennix @shuijing198799 PTAL

cofyc · 2019-09-11T09:02:10Z

tests/cmd/fault-trigger/main.go

@@ -43,7 +45,7 @@ func main() {
 	logs.InitLogs()
 	defer logs.FlushLogs()

-	mgr := manager.NewManager()
+	mgr := manager.NewManager(vmManager)


better not to allow invalid --vm-manager value, we can initialize vm manager and handle errors here, e.g.

var vmMgr VMManager if vmManager == "qm" { vmMgr = NewQMManager() } else if vmManager == "virsh" { vmMgr = NewVirshManager() } else { // fatal error } mgr := manager.NewManager(vmMgr)

if some users configured an invalid value, but our program still works, this will confuse people because they don't know what virtual manager we use from the command-line flags unless they know implementation details

…db-operator into agent-support-qm

xiaojingchen · 2019-09-12T04:20:59Z

/run-e2e-in-kind

cofyc · 2019-09-12T05:09:18Z

tests/pkg/fault-trigger/manager/manager.go

+	} else if vmManagerName == "virsh" {
+		vmManager = &VirshVMManager{}
+	} else {
+		slack.NotifyAndPanic(fmt.Errorf("stability test have not supported the vm manager:[%s],please choose [qm] or [virsh].", vmManagerName))


in fault-trigger, slack.Webhook endpoint is not provided. IMO a simple fatal error is enough.

xiaojingchen · 2019-09-12T09:48:51Z

/run-e2e-in-kind

cofyc · 2019-09-18T05:27:17Z

this lgtm but I cannot test it locally
what's the progress, have you tested it successfully in the stability environment?

…db-operator into agent-support-qm

xiaojingchen · 2019-09-24T03:47:20Z

@cofyc this pr has been tested pass in stability env. now it is still running in stability env.

cofyc

LGTM

xiaojingchen · 2019-09-27T09:47:16Z

@cofyc @weekface @aylei PTAL again

weekface · 2019-09-27T09:49:51Z

Have you run this PR on the Virsh manager cluster?

xiaojingchen · 2019-09-27T10:13:50Z

it has run in stability env for serval days，except the apiserver fault trigger case.
but the apiserver fault trigger case has passed separately.

xiaojingchen · 2019-09-27T10:18:04Z

@weekface it has not run in stability env which usevirsh to manage vm. and now I lack the stability env base virsh

weekface · 2019-09-27T10:23:02Z

You can run it at 149 env.

weekface

LGTM

cofyc

LGTM

cofyc · 2019-10-12T02:33:47Z

/run-e2e-in-kind

cofyc · 2019-10-12T02:35:25Z

@xiaojingchen CI failed

xiaojingchen · 2019-10-12T03:00:27Z

/run-e2e-in-kind

…db-operator into agent-support-qm

xiaojingchen · 2019-10-12T03:02:49Z

/run-e2e-in-kind

weekface

LGTM

xiaojingchen · 2019-10-12T04:17:04Z

@cofyc PTAL again

(cherry picked from commit 4d2dd0c)

(cherry picked from commit 4d2dd0c) #896

refactor fault trigger

689aa7c

cofyc reviewed Sep 10, 2019

View reviewed changes

weekface reviewed Sep 10, 2019

View reviewed changes

xiaojingchen added 2 commits September 11, 2019 16:10

fix config.yaml

8a7e6a1

address comment

f836536

xiaojingchen changed the title ~~[WIP]tests: refactor fault trigger~~ tests: refactor fault trigger Sep 11, 2019

Merge branch 'master' into agent-support-qm

8648f7e

cofyc reviewed Sep 11, 2019

View reviewed changes

xiaojingchen added 2 commits September 12, 2019 12:17

address comment

1721243

Merge branch 'agent-support-qm' of https://github.com/xiaojingchen/ti…

7d1b2e1

…db-operator into agent-support-qm

fix unit tests

7b79f2c

cofyc reviewed Sep 12, 2019

View reviewed changes

fix bugs

b5bc4e0

xiaojingchen and others added 4 commits September 16, 2019 16:01

fix

c6449e6

Merge branch 'master' into agent-support-qm

82cafca

fix compatibility bug

14e76a8

Merge branch 'master' into agent-support-qm

f39f241

xiaojingchen added 2 commits September 18, 2019 16:32

add test name

ac4070a

Merge branch 'agent-support-qm' of https://github.com/xiaojingchen/ti…

b09c8b1

…db-operator into agent-support-qm

cofyc previously approved these changes Sep 24, 2019

View reviewed changes

xiaojingchen added 5 commits September 25, 2019 12:15

refactor fault trigger

a3faa0e

fix config.yaml

54709e4

address comment

68ddae7

address comment

5753e98

fix unit tests

2cd1da6

Merge branch 'master' into agent-support-qm

f9dbde8

Merge branch 'master' into agent-support-qm

6afcd3b

tennix added the test/stability stability tests label Oct 11, 2019

fix

ed5c31a

weekface previously approved these changes Oct 11, 2019

View reviewed changes

weekface added the status/LGTM1 label Oct 11, 2019

Merge branch 'master' into agent-support-qm

556238a

cofyc approved these changes Oct 12, 2019

View reviewed changes

xiaojingchen added 2 commits October 12, 2019 10:57

fix lint error

04af526

Merge branch 'master' into agent-support-qm

16e083e

cofyc previously approved these changes Oct 12, 2019

View reviewed changes

Merge branch 'agent-support-qm' of https://github.com/xiaojingchen/ti…

1f4bd01

…db-operator into agent-support-qm

xiaojingchen dismissed stale reviews from cofyc and weekface via 1f4bd01 October 12, 2019 03:02

weekface mentioned this pull request Oct 12, 2019

test: ensure pods unaffected when upgrading #955

Merged

weekface approved these changes Oct 12, 2019

View reviewed changes

cofyc approved these changes Oct 12, 2019

View reviewed changes

cofyc merged commit 4d2dd0c into pingcap:master Oct 12, 2019

weekface pushed a commit to weekface/tidb-operator that referenced this pull request Oct 14, 2019

tests: refactor fault trigger (pingcap#896)

3671b48

(cherry picked from commit 4d2dd0c)

weekface added a commit that referenced this pull request Oct 14, 2019

tests: refactor fault trigger (#896) (#1009)

43d09ca

(cherry picked from commit 4d2dd0c) #896

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: refactor fault trigger #896

tests: refactor fault trigger #896

xiaojingchen commented Sep 10, 2019 •

edited

Loading

cofyc Sep 10, 2019

weekface Sep 10, 2019

xiaojingchen commented Sep 11, 2019

cofyc Sep 11, 2019 •

edited

Loading

xiaojingchen commented Sep 12, 2019

cofyc Sep 12, 2019

xiaojingchen commented Sep 12, 2019

cofyc commented Sep 18, 2019

xiaojingchen commented Sep 24, 2019 •

edited

Loading

cofyc left a comment

xiaojingchen commented Sep 27, 2019

weekface commented Sep 27, 2019

xiaojingchen commented Sep 27, 2019 •

edited

Loading

xiaojingchen commented Sep 27, 2019 •

edited

Loading

weekface commented Sep 27, 2019

weekface left a comment

cofyc left a comment

cofyc commented Oct 12, 2019

cofyc commented Oct 12, 2019

xiaojingchen commented Oct 12, 2019

xiaojingchen commented Oct 12, 2019

weekface left a comment

xiaojingchen commented Oct 12, 2019

tests: refactor fault trigger #896

tests: refactor fault trigger #896

Conversation

xiaojingchen commented Sep 10, 2019 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Does this PR introduce a user-facing change?:

cofyc Sep 10, 2019

Choose a reason for hiding this comment

weekface Sep 10, 2019

Choose a reason for hiding this comment

xiaojingchen commented Sep 11, 2019

cofyc Sep 11, 2019 • edited Loading

Choose a reason for hiding this comment

xiaojingchen commented Sep 12, 2019

cofyc Sep 12, 2019

Choose a reason for hiding this comment

xiaojingchen commented Sep 12, 2019

cofyc commented Sep 18, 2019

xiaojingchen commented Sep 24, 2019 • edited Loading

cofyc left a comment

Choose a reason for hiding this comment

xiaojingchen commented Sep 27, 2019

weekface commented Sep 27, 2019

xiaojingchen commented Sep 27, 2019 • edited Loading

xiaojingchen commented Sep 27, 2019 • edited Loading

weekface commented Sep 27, 2019

weekface left a comment

Choose a reason for hiding this comment

cofyc left a comment

Choose a reason for hiding this comment

cofyc commented Oct 12, 2019

cofyc commented Oct 12, 2019

xiaojingchen commented Oct 12, 2019

xiaojingchen commented Oct 12, 2019

weekface left a comment

Choose a reason for hiding this comment

xiaojingchen commented Oct 12, 2019

xiaojingchen commented Sep 10, 2019 •

edited

Loading

cofyc Sep 11, 2019 •

edited

Loading

xiaojingchen commented Sep 24, 2019 •

edited

Loading

xiaojingchen commented Sep 27, 2019 •

edited

Loading

xiaojingchen commented Sep 27, 2019 •

edited

Loading