[Usage] Fix flaky UI Counters test #100979

Bamieh · 2021-05-31T07:48:23Z

Increase the duration to wait for ES indices to refresh after SO increments from 5 to 8 seconds. 8 seconds is long enough to remove flakiness. Already fixed in other similar telemetry test cases.

Ran this test 1000 times locally with no flakiness.
Ran the test 1000 times on the kibana flaky test runner (https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/1599/)

closes #93159 and #98240

elasticmachine · 2021-05-31T07:48:25Z

Pinging @elastic/kibana-core (Team:Core)

elasticmachine · 2021-05-31T07:48:25Z

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

pgayvallet · 2021-05-31T13:27:11Z

Ran this test 1000 times locally with no flakiness.

Could you run it against the flaky test runner? Such flakiness is often not reproducible locally as much as on CI

mshustov · 2021-05-31T14:10:54Z

Could you run it against the flaky test runner?

@Bamieh just in case, it's here

Bamieh · 2021-05-31T14:14:18Z

@mshustov thanks! I was wondering if that is actually a thing 😅

Bamieh · 2021-06-01T11:07:18Z

@pgayvallet @mshustov Flaky test runner passed https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/1599/

Bamieh · 2021-06-01T11:07:52Z

@elasticmachine merge upstream

mshustov · 2021-06-01T12:50:35Z

test/api_integration/apis/ui_counters/ui_counters.ts

@@ -32,7 +32,7 @@ export default function ({ getService }: FtrProviderContext) {
      .expect(200);

    // wait for SO to index data into ES
-    await new Promise((res) => setTimeout(res, 5 * 1000));
+    await new Promise((res) => setTimeout(res, 8 * 1000));


Can we retry requesting /api/saved_objects/_find?type=usage-counters until it's succeeded instead of relying on a manual delay?

That is doable although the delay pattern is all across kibana tests (200+ places)

Agree, but it doesn't mean it's a good thing.

it leads to false positives due to changes in Kibana internals (what you are fixing right now)

it introduces unnecessary delays in the execution flow. Maybe the condition is true after 2sec?

it's not clear how to pick a correct value, devs tend to copy-paste them

I completely agree. What i meant is that we might need to resolve it across all tests rather than just this one test. I've created a helper util (with tests) to retry assertions. We can move it outside the tests/.../telemetry folder in a separate PR later on. Let me know what you think.

We can move it outside the tests/.../telemetry folder in a separate PR later on.

ok then. Let's create an issue for Core-related tests.

…counters_test

… into usage/fix_ui_counters_test

test/api_integration/apis/index.ts

test/api_integration/apis/telemetry/utils/try_assertion_until.ts

kibanamachine · 2021-06-03T14:44:53Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #128894 succeeded a8f8599
💚 Build #128296 succeeded 9dda838
💔 Build #128289 failed 822180d
💚 Build #128132 succeeded dbe0bbb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-06-03T14:50:35Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Ahmad Bamieh <ahmadbamieh@gmail.com>

increase to delay to 8 seconds

dbe0bbb

Bamieh added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:Telemetry Team:KibanaTelemetry labels May 31, 2021

Bamieh requested a review from a team May 31, 2021 07:48

Bamieh added v7.14.0 v8.0.0 auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes labels May 31, 2021

[TEMP] ONLY UI COUNTERS TEST FILE

822180d

Update index.ts

64c029b

Merge branch 'master' into usage/fix_ui_counters_test

9dda838

mshustov reviewed Jun 1, 2021

View reviewed changes

Bamieh added 4 commits June 2, 2021 16:21

Merge branch 'master' of github.com:elastic/kibana into usage/fix_ui_…

b4e6f6f

…counters_test

Merge branch 'master' of github.com:elastic/kibana into usage/fix_ui_…

229aa3e

…counters_test

retry assertion function

651101e

Merge branch 'usage/fix_ui_counters_test' of github.com:Bamieh/kibana…

04b583c

… into usage/fix_ui_counters_test

Bamieh requested a review from mshustov June 3, 2021 12:07

Bamieh commented Jun 3, 2021

View reviewed changes

test/api_integration/apis/index.ts Outdated Show resolved Hide resolved

mshustov reviewed Jun 3, 2021

View reviewed changes

test/api_integration/apis/telemetry/utils/try_assertion_until.ts Outdated Show resolved Hide resolved

mshustov approved these changes Jun 3, 2021

View reviewed changes

Bamieh added 2 commits June 3, 2021 15:36

self codereview

a8f8599

revert and use retry common util

2e8d126

uncomment test files

225c783

Bamieh merged commit de85036 into elastic:master Jun 3, 2021

Bamieh deleted the usage/fix_ui_counters_test branch June 3, 2021 14:47

kibanamachine mentioned this pull request Jun 3, 2021

[7.x] [Usage] Fix flaky UI Counters test (#100979) #101284

Merged

kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Jun 3, 2021

[Usage] Fix flaky UI Counters test (elastic#100979)

72d2272

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Bamieh mentioned this pull request Jun 3, 2021

Failing test: API Integration Tests.test/api_integration/apis/ui_counters/ui_counters·ts - apis UI Counters UI Counters API stores ui counter events in usage counters savedObjects #98240

Closed

kibanamachine added a commit that referenced this pull request Jun 3, 2021

[Usage] Fix flaky UI Counters test (#100979) (#101284)

4238ecf

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Ahmad Bamieh <ahmadbamieh@gmail.com>

lukeelmers removed the Team:KibanaTelemetry label Oct 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Usage] Fix flaky UI Counters test #100979

[Usage] Fix flaky UI Counters test #100979

Bamieh commented May 31, 2021 •

edited

Loading

elasticmachine commented May 31, 2021

elasticmachine commented May 31, 2021

pgayvallet commented May 31, 2021

mshustov commented May 31, 2021

Bamieh commented May 31, 2021

Bamieh commented Jun 1, 2021 •

edited

Loading

Bamieh commented Jun 1, 2021

mshustov Jun 1, 2021

Bamieh Jun 1, 2021

mshustov Jun 3, 2021

Bamieh Jun 3, 2021

mshustov Jun 3, 2021

kibanamachine commented Jun 3, 2021

kibanamachine commented Jun 3, 2021

[Usage] Fix flaky UI Counters test #100979

[Usage] Fix flaky UI Counters test #100979

Conversation

Bamieh commented May 31, 2021 • edited Loading

elasticmachine commented May 31, 2021

elasticmachine commented May 31, 2021

pgayvallet commented May 31, 2021

mshustov commented May 31, 2021

Bamieh commented May 31, 2021

Bamieh commented Jun 1, 2021 • edited Loading

Bamieh commented Jun 1, 2021

mshustov Jun 1, 2021

Choose a reason for hiding this comment

Bamieh Jun 1, 2021

Choose a reason for hiding this comment

mshustov Jun 3, 2021

Choose a reason for hiding this comment

Bamieh Jun 3, 2021

Choose a reason for hiding this comment

mshustov Jun 3, 2021

Choose a reason for hiding this comment

kibanamachine commented Jun 3, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Jun 3, 2021

💚 Backport successful

Bamieh commented May 31, 2021 •

edited

Loading

Bamieh commented Jun 1, 2021 •

edited

Loading