Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue for restructure tests #26022

Closed
53 tasks done
tisonkun opened this issue Jul 7, 2021 · 24 comments
Closed
53 tasks done

Tracking issue for restructure tests #26022

tisonkun opened this issue Jul 7, 2021 · 24 comments
Assignees
@tisonkun
Labels
component/test good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. type/enhancement
Projects
Robust test

Comments

@tisonkun
Copy link
Contributor

tisonkun commented Jul 7, 2021 •
edited

Brief Motivation

This issue tracks the efforts to restructure TiDB tests, focusing on unit tests and integration tests, especially on separating integration_test.go as well as unifying test infrastructure in testify.

For the detailed design, see also:

Join force

We are currently on the first phase on the effort and focus on migrating tests to testify. If you want to participant the movement, you can directly comment under any open issues below without assignee to require assignment.

You can take a look at #26375 as an example to do the migration.

Note

Before writing code, please take a look at PRs merged already and keep consistent code style.

Migration

Kits

Reference
@tisonkun tisonkun mentioned this issue Jul 7, 2021
Merged
@tisonkun tisonkun added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. component/test labels Jul 9, 2021
@github-actions github-actions bot added this to TODO/Help Wanted in Robust test Jul 9, 2021
@jyz0309
Copy link
Contributor

jyz0309 commented Jul 10, 2021

A little question, if we use testify to instead pingcap/check, maybe we need to restructure the /utils/testkit first? It seem that we use TestKit.MustExec to exec sql in TiDB test, and MustExec need to use pingcap/check.

@tisonkun
Copy link
Contributor Author

tisonkun commented Jul 10, 2021 •
edited

@jyz0309 Yes of course. That is in the plan and we will create testkit with go testing & testify. We use go-check for matchers almost and it won't be too complex.

Tracked as #26108

@tisonkun
Copy link
Contributor Author

tisonkun commented Jul 10, 2021 •
edited

@jyz0309 BTW #26102 & #26103 are still free to be assigned. If you want to take one of them, feel free to comment there.

This was referenced Jul 10, 2021
Merged
Closed
Closed
Closed
@tisonkun
Copy link
Contributor Author

In an offline discussion with @tiancaiamao it seems we'd better consider test timeout in this effort also.

gocheck provides a check.timeout mechanism but the current state seems unsatisfied (why?). golang itself doesn't provides timeout per function.

After a little investigation, I write a helper for this requirement

func RunWithTimeout(t *testing.T, d time.Duration, f func(*testing.T)) {
	timeout := time.After(d)
	done := make(chan bool)
	go func() {
		defer func() {
			done <- true
		}()
		f(t)
	}()
	select {
	case <-timeout:
		t.Fatalf("timeout: %s", d.String())
	case <-done:
	}
}

... which is used as

func TestName(t *testing.T) {
	testkit.RunWithTimeout(t, 1 * time.Second, func(t *testing.T) {
		require.Equal(t, 2, 2)
		time.Sleep(2 * time.Second)
	})
}

.. one more level of ident but works well.

However, @tiancaiamao means the original issue is that a panic from a previous test cause the whole test hang. I don't know what does it mean, since a main goroutine panic will fail the test immediately.

@tisonkun tisonkun mentioned this issue Jul 18, 2021
Closed
22 tasks
@tiancaiamao
Copy link
Contributor

the original issue is that a panic from a previous test cause the whole test hang. I don't know what does it mean

It's a problem of gocheck, it spawn new goroutines to run the test cases, so a panic does not make the process exit immediately. @tisonkun

@tisonkun tisonkun mentioned this issue Jul 19, 2021
Merged
@PragmaTwice
Copy link
Contributor

Hi @tisonkun, I'd like to migrate some tests under utils pkg, is there any task still free to assign?

@tisonkun
Copy link
Contributor Author

@PragmaTwice most of util/xxx are free to assign, you may just jump to subtasks listed above.

@PragmaTwice PragmaTwice mentioned this issue Jul 21, 2021
Merged
@tisonkun tisonkun mentioned this issue Jul 22, 2021
Open
9 tasks
@PragmaTwice PragmaTwice mentioned this issue Jul 22, 2021
Merged
@beyoung
Copy link
Contributor

beyoung commented Jul 23, 2021

Hi @tisonkun ,I'd like to migrate util/plancodec pkg , which is not listed above, please create a subtask and assign to me . Thx.

@tisonkun
Copy link
Contributor Author

Hi @tisonkun ,I'd like to migrate util/plancodec pkg , which is not listed above, please create a subtask and assign to me . Thx.

Aha! That's it. Let me create one for you.

@tisonkun tisonkun mentioned this issue Jul 26, 2021
Closed
@tiancaiamao tiancaiamao mentioned this issue Jul 30, 2021
Merged
12 tasks
@tisonkun tisonkun pinned this issue Jul 31, 2021
@tisonkun
Copy link
Contributor Author

tisonkun commented Aug 3, 2021

@tisonkun It seems failpoints conflict with each others. Let me batch domain_test to run in serial.
@xhebox failpoints and global config are both serial-safe, maybe mention it in the tracking issue.

#26752

@feitian124 feitian124 mentioned this issue Aug 4, 2021
Closed
@kevgeo kevgeo mentioned this issue Aug 5, 2021
Closed
@maxshuang
Copy link

maxshuang commented Sep 22, 2021 via email

@w169q169
Copy link
Contributor

@tisonkun You idea is same as me, tools could do easy work and more complex work need be checked manually .
Review is always necessary, bad/corner cases that I dont know such as c.Check(len(bindData.Bindings), Equals, 0) and c.Assert(err, IsNil), I have added feature to fix the tools .
image

@feitian124 feitian124 mentioned this issue Oct 5, 2021
Merged
@ti-chi-bot ti-chi-bot mentioned this issue Oct 12, 2021
Merged
@tisonkun tisonkun mentioned this issue Oct 26, 2021
Closed
@tisonkun tisonkun mentioned this issue Nov 6, 2021
Closed
12 tasks
@hawkingrei hawkingrei mentioned this issue Nov 15, 2021
Merged
1 task
@tisonkun
Copy link
Contributor Author

As mentioned in #30692, we decided to run all tests in serial.

Thanks to @tangenta for driving the decision and I've updated the description in this issue. Thanks for all of your participants.

@dveeden
Copy link
Contributor

dveeden commented Dec 27, 2021

@tisonkun I assume https://pingcap.github.io/tidb-dev-guide/get-started/write-and-run-unit-tests.html#parallel needs updating? Should that section be replaced by an explanation about why we want to run things in serial? or keep it?

@tisonkun tisonkun mentioned this issue Jan 6, 2022
Closed
@hawkingrei hawkingrei mentioned this issue Jan 12, 2022
Merged
12 tasks
@hawkingrei hawkingrei mentioned this issue Jan 20, 2022
Merged
12 tasks
@tisonkun tisonkun mentioned this issue Feb 28, 2022
Merged
@kang8 kang8 mentioned this issue Mar 1, 2022
Closed
@tisonkun
Copy link
Contributor Author

So...

As #33296 got merged, we have new subtasks to resolve.

@tisonkun tisonkun mentioned this issue Mar 25, 2022
Closed
This was referenced Apr 11, 2022
Merged
Merged
@JmPotato JmPotato mentioned this issue Apr 15, 2022
Closed
85 tasks
@tisonkun
Copy link
Contributor Author

Closed as all subtasks resolved. Thanks for everyone help in this thread!

Robust test automation moved this from TODO/Help Wanted to Done May 17, 2022
@tisonkun tisonkun unpinned this issue May 17, 2022
@tisonkun tisonkun mentioned this issue May 18, 2022
Closed
8 tasks
@tiancaiamao tiancaiamao mentioned this issue May 19, 2022
Merged
12 tasks
@tisonkun tisonkun mentioned this issue May 30, 2022
Closed
@tisonkun tisonkun self-assigned this Aug 16, 2022
@tisonkun tisonkun mentioned this issue Sep 14, 2022
Closed
2 tasks
@tisonkun tisonkun mentioned this issue Mar 12, 2024
Closed
2 tasks
@tisonkun tisonkun mentioned this issue Apr 10, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tisonkun tisonkun

Labels
component/test good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. type/enhancement
Projects
Robust test
  
Done
Milestone
No milestone
Development

No branches or pull requests
10 participants
@dveeden @tiancaiamao @feitian124 @beyoung @jiaxiang521 @w169q169 @maxshuang @tisonkun @PragmaTwice @jyz0309