[Improvement] Make FTR tests independent #1167
Labels
enhancement
New feature or request
technical debt
If not paid, jeapardizes long-term success and maintainability of the repository.
test:functional
Description
Executing
yarn test:ftr
could potentially lead to failures (or unexpected consequences) in a series of unrelated functional tests because an earlier test failed (or even sometimes succeeded) and prevented the test from closing in a "zero state".For example, test group 1 could have the following test files: TestSettings, TestOverviewPage, TestDiscoverPage. All the tests would require returning to the Home page in the "clean up" step and require to start in the Home page. If TestOveragePage ends up not rendering properly, the test runner can fail and not execute the "clean up" step thus not putting the application the Home page and TestDiscoverPage failed because it was trying to use the test runner to navigate to the DiscoverPage from the Home page.
That is for failures, but for non-failures, TestSettings will impact TestOverviewPage and TestDiscoverPage, and TestOverviewPage will impact TestDiscoverPage. This becomes problematic if we want to test the welcomeScreen because if the application is not in a "zero state" with no actual docs in the system index then it will not show that welcomeScreen. So if TestSettings adding docs to the system index then it would prevent a later test from seeing that welcomeScreen if there is a specific test for that. Putting the test first would still not be desired because we should be able to run any file without worrying about previous runs.
Expected Result
Functional tests are independent. If the test requires going to Settings and Overview page to test the Discover page then the Discover test should execute as a setup step.
Pros
Tests are more isolated and granular to figure out the right component that failed. So one failure doesn't cause a failure in a suite of tests.
Cons
Longer execution time
The text was updated successfully, but these errors were encountered: