Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Functional] Fixes flakiness on TSVB indexpattern switch #78754

Merged
merged 2 commits into from
Oct 1, 2020
Merged

[Functional] Fixes flakiness on TSVB indexpattern switch #78754

merged 2 commits into from
Oct 1, 2020

Conversation

stratoula
Copy link
Contributor

@stratoula stratoula commented Sep 29, 2020 •
edited
Loading

Summary

Closes #76190
Closes #71979
This functional test fails a lot both on cloud and on regular test runners. The problem is that while it adds the new index pattern the time fields need some time to be loaded so it falls on a race condition as it tries to click on a time option that hasn't been loaded yet.

I have changed the implementation by instead of waiting 3sec (which is a magic number), it waits for the options on the timefield dropdown to be loaded. When there are options it selects the timestamp. I have run it on normal test runner 100 times and 15 times on cloud. It seems that it doesn't fail.

https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/858/
https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/865/
https://internal-ci.elastic.co/job/elastic+estf-cloud-kibana-flaky-test-runner/137/
https://internal-ci.elastic.co/job/elastic+estf-cloud-kibana-flaky-test-runner/144/
https://internal-ci.elastic.co/job/elastic+estf-cloud-kibana-flaky-test-runner/145/

@stratoula stratoula changed the title Change implementation on TSVB functional when testing the indexPatter… [Functional]Fixes flakiness on TSVB indexPattern switch Sep 29, 2020
@stratoula stratoula changed the title [Functional]Fixes flakiness on TSVB indexPattern switch [Functional]Fixes flakiness on TSVB indexpattern switch Sep 29, 2020
@stratoula stratoula changed the title [Functional]Fixes flakiness on TSVB indexpattern switch [Functional] Fixes flakiness on TSVB indexpattern switch Sep 29, 2020
@stratoula stratoula added Feature:TSVB TSVB (Time Series Visual Builder) release_note:skip Skip the PR/issue when compiling release notes v8.0.0 v7.10.0 labels Sep 30, 2020
@stratoula stratoula marked this pull request as ready for review September 30, 2020 11:59
@stratoula stratoula added the failed-test A test failure on a tracked branch, potentially flaky-test label Sep 30, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-test-triage (failed-test)

@stratoula stratoula added the Team:Visualizations Visualization editors, elastic-charts and infrastructure label Sep 30, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app (Team:KibanaApp)

@stratoula
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

dmlemeshko
dmlemeshko approved these changes Oct 1, 2020
View reviewed changes
Copy link
Member

@dmlemeshko dmlemeshko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for not giving up on this test, it's been flaky since its creation (well, TSVB reliability is responsible for it as well)
LGTM

test/functional/page_objects/visual_builder_page.ts
Comment on lines +457 to +462
await retry.waitFor('combobox options loaded', async () => {
const options = await comboBox.getOptions('metricsIndexPatternFieldsSelect');
log.debug(`-- optionsCount=${options.length}`);
return options.length > 0;
});
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we need a retry here? I wonder if there is a chance that option list will be updated without field value being re-typed?

Looks like openOptionsList is not reopening list, if it is already displayed.

It's not critical if it works stable, just want to point out that using nested retries might slow things down.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think it is as await comboBox.getOptions('metricsIndexPatternFieldsSelect'); can return 0. The problem is that the metricsIndexPatternFieldsSelect dropdown has a delay on filling in with the options so here the retry helps to get the options and when there are > 0 it will continue to the rest of the test

@stratoula stratoula merged commit 36814aa into elastic:master Oct 1, 2020
stratoula added a commit to stratoula/kibana that referenced this pull request Oct 1, 2020
@stratoula @elasticmachine
Change implementation on TSVB functional when testing the indexPatter…
50f9a9e 
…n switch (elastic#78754)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@stratoula stratoula mentioned this pull request Oct 1, 2020
Merged
stratoula added a commit that referenced this pull request Oct 1, 2020
@stratoula @elasticmachine
Change implementation on TSVB functional when testing the indexPatter…
7d34537 
…n switch (#78754) (#79089)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dmlemeshko dmlemeshko dmlemeshko approved these changes

Assignees
No one assigned
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Feature:TSVB TSVB (Time Series Visual Builder) release_note:skip Skip the PR/issue when compiling release notes Team:Visualizations Visualization editors, elastic-charts and infrastructure v7.10.0 v8.0.0
Projects
None yet
Milestone
No milestone
4 participants
@stratoula @elasticmachine @kibanamachine @dmlemeshko