tests: add a test case mark tool

[mahjonp]

What problem does this PR solve?

This PR adds a check to verify that incremental test cases are annotated with the associated Feature ID or Issue ID. For example, a test case should have the following form:

func TestBindingWithoutCharset(t *testing.T) {
    // Used to mark a Feature
    marker.As(t, maker.Feature, "FD-1024", "balabala")
    // Or no FD is assigned
    marker.As(t, maker.Feature, maker.NoID)
    // Or used to mark an Issue
    marker.As(t, maker.Issue, 1234)
    ...
}

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[TennyZhuang]

The implementation looks stupid enough, and introduce too many invasions for developers. If you are interested in some metrics about fine-grained test coverage, why not associate the incremental tests with the commit, and related issue?
An incremental test can also be found automatically by scanning all tests before and after a specified commit, then diff them.

[mahjonp]

The implementation looks stupid enough, and introduce too many invasions for developers. If you are interested in some metrics about fine-grained test coverage, why not associate the incremental tests with the commit, and related issue? An incremental test can also be found automatically by scanning all tests before and after a specified commit, then diff them.

Sorry for the invasions to the developers, what you said makes sense. I will look into whether the current R&D process is complete and if it can index a complete set of PRs from the feature id. @TennyZhuang

[mahjonp]

The implementation looks stupid enough, and introduce too many invasions for developers. If you are interested in some metrics about fine-grained test coverage, why not associate the incremental tests with the commit, and related issue? An incremental test can also be found automatically by scanning all tests before and after a specified commit, then diff them.

Sorry for the invasions to the developers, what you said makes sense. I will look into whether the current R&D process is complete and if it can index a complete set of PRs from the feature id. @TennyZhuang

By the way, this PR is planned to use nogo config to control the directories or files that are checked in alpha testing (for example starting from the sql team first), collect feedback and improve during the practice, and will not be released all at once. It is considered to minimize the intrusion to developers.

[mahjonp]

Suggested change

".*_test\\.go$": "ignore test code"

will remove later and select some dirs for alpha testing

[mahjonp]

After survey, I found that the current R&D process is unable to link the Feature ID(create on Jira) to a GitHub PR or issue, and that there is no feature id information attached to the issue or PR. Therefore, adding a marker may be the only current solution, at least for marking FD-$ID for feature development PR 😰

[mahjonp]

/rebuild

[mahjonp]

/run-build

[mahjonp]

/run-build

[mahjonp]

/run-build

[mahjonp]

/run-all-tests

[TennyZhuang]

Please also update CONTRIBUTION.md

Bug fixes usually come with tests. With the help of continuous integration test, patches can be easy to review. Please update the unit tests so that they catch the bug! Please check example #2808.

A community contributor can't add a test now, so please remove these lines.

Some examples of "Architecture" improvements:
Improving test coverage.

ditto

Make commits of logical units and add test case if the change fixes a bug or adds new functionality.

ditto

---

Another simple solution: Please remove the fake CONTRIBUTING.md.

[mahjonp]

close because of controversy