Fix flaky TestCafe OV polling test in `DeviceChallengePollView_spec` #3654

denysoblohin-okta · 2024-06-14T18:54:12Z

Description:

The fix consists of 2 parts:

Need to comment skipped tests. For some reason (bug in TestCafe 1.x?) mocks used in test.skip.requestHooks(xxx) interferes with mocks in other tests - other tests in spec can use mocks from skipped test
Promisified $.ajax in BaseOktaVerifyChallengeView - migrated from jQuery's Promise-like done/fail to native then/catch. With old approach, probe and challenge requests are not chained correctly (et. probing port 6512 can be started in parallel with challenging port 6511). With native Promise it's always a chain probe 1 -> challenge 1 -> probe 2 -> challenge 2 -> ...

Notes:

Tested the fix 10 times on Bacon and got constant green testcafe task
Tried to upgrade TestCafe to 3.x to eliminate need to comment skipped tests, but it requires upgrading Chrome version and causes other issues, so postponed..

PR Checklist

Have you verified the basic functionality for this change?
Did you add tests, as appropriate, following our Automated Test guidelines?
Did you follow our Security Best Practices?
Did you verify the change by running downstream monolith artifact? (Provide link to build below)
Does this PR include noticeable changes to the UI? (If yes, attach screenshots/video below)

Issue:

OKTA-715718

Reviewers:

Screenshot/Video:

Downstream Monolith Build:

test/testcafe/spec/DeviceChallengePollView_spec.js

lesterchoi-okta · 2024-06-18T15:24:39Z

src/v2/view-builder/internals/BaseOktaVerifyChallengeView.js

@@ -14,7 +14,13 @@ const request = (opts) => {
    method: 'GET',
    contentType: 'application/json',
  }, opts);
-  return $.ajax(ajaxOptions);
+  return new Promise((resolve, reject) => {


Explain why/how this helps

@shuowu-okta did you have a specific behavior-change related concern w.r.t. Promisifying this?

In current code Promises are mixed with jQuery's Deferred (line 159 let probeChain = Promise.resolve(); uses Promise, $.ajax returns Deferred and done/fail APIs are used)

Tests expect that requests are performed in chain: probe port 2000 -> challenge 2000 (if probe OK) -> probe 6511 -> challenge 6511 -> probe 6512 -> challenge 6512 -> probe 6513 -> challenge 6513
With current code it's not always true, eg. probe 6511 can run in parallel with challenge 2000
So eg. this check

okta-signin-widget/test/testcafe/spec/DeviceChallengePollView_spec.js

Line 435 in 054cb76

await t.expect(loopbackChallengeErrorLogger.contains(record => record.request.url.match(/6512|6513/))).eql(false);

can fail

Changing the base xhr approach may cause unexpected issue for legacy browsers. As the goal is to fix flaky bacon test, I propose to find another way that fix against test directly, for example, migrate test from e2e to unit level (testcafe -> jest).

See latest commit that uses $.Deferred instead of Promise and then+catch API instead of done+fail for proper chaining.
The recommended API method to use chaining is then: https://api.jquery.com/deferred.then/

yannongli-okta · 2024-06-20T17:59:47Z

src/v2/view-builder/internals/BaseOktaVerifyChallengeView.js

    };

-    let probeChain = Promise.resolve();
+    let probeChain = $.Deferred().resolve();


will this deferred cause any degradation of UX like slower authentication speed?

With this change the behaviour will be same as in Gen3 - probe/challenge calls will be running not in parallel, but in chain (probing/challenging for port 2 should wait for probing/challenging of port 1 to complete).
This can be slower.
But such behaviour in expected judging from code and tests.

If it's desired to have parallel requests, then tests should be rewritten to reflect this.
For example, this line should be removed:

okta-signin-widget/test/testcafe/spec/DeviceChallengePollView_spec.js

Line 430 in c07d759

await t.expect(loopbackChallengeErrorLogger.contains(record => record.request.url.match(/6512|6513/))).eql(false);

cc @shuowu-okta @Luxu2-Okta

After talking with the team, we would like to fix it, to make V2 behave similar like V3, there shouldn't be a performance concern.
We can be more cautious by using a new flag from server, but this might be an overkill

We typically isolate higher-risk changes in a patch bump on the tip so it can be easily rolled back. I think this can target 7.22.1

the $.Deferred().resolve() is the jquery's promise, not sure about the very low level, but jquery promise has the .done method (similar as .then), while es6 promise doesn't.
since all the probing will use jquery $.ajax method, so I guess using $.Deferred().resolve() and return the rejected promise in onFailure will make it serial, but I will leave it to @denysoblohin-okta to confirm

How does using $.Deferred().resolve() differ from Promise.resolve()

First one returns $.Deferred object instead of Promise. This will change probeChain from a chain of Promises to a chain of Deferred. Deferred is a "thenable" object, but not a real promise.

how does it cause the probe chain to become serial?

In current code native Promises are mixed with jQuery's Deferred:

okta-signin-widget/src/v2/view-builder/internals/BaseOktaVerifyChallengeView.js

Line 166 in f37fdb3

let probeChain = Promise.resolve();

okta-signin-widget/src/v2/view-builder/internals/BaseOktaVerifyChallengeView.js

Line 22 in f37fdb3

return $.ajax(ajaxOptions);

($.ajax returns $.Deferred)

probeChain uses native Promises with then/catch API. But doProbing() returns $.Deferred and uses done/fail API.
This is a problem. To achieve proper chaining we need to change done/fail API to then/catch.

Actually we can leave Promise.resolve() instead of $.Deferred().resolve(), then we still have mixing of Promise and Deferred, but they can play well together with then/catch API.

Rebased PR onto 7.21

just curious, this will target 7.21.2 in the 2024.09.0?

Actually we can leave Promise.resolve() instead of $.Deferred().resolve()

Yeah, I think now that the Promise API is standardized and canonically polyfilled we should try to use Promise when we can since it's better understood and well-documented (MDN).

Luxu2-Okta

LGTM, if we want to be very cautious, we can put this fix behind a FF or KS, but this maybe an overkill

Luxu2-Okta

LGTM!

lesterchoi-okta

Hold for 7.22.1 release

denysoblohin-okta force-pushed the od-fix-flacky-poll-tests branch from 2bba797 to 268edc4 Compare June 14, 2024 18:55

denysoblohin-okta marked this pull request as ready for review June 14, 2024 19:54

denysoblohin-okta requested review from glenfannin-okta, jaredperreault-okta, leemchale-okta, lesterchoi-okta, robertdamphousse-okta, shawyu-okta, shuowu-okta, trevoring-okta and vijetmahabaleshwar-okta as code owners June 14, 2024 19:54

Luxu2-Okta mentioned this pull request Jun 14, 2024

Okta 719439 https loopback #3650

Merged

5 tasks

denysoblohin-okta force-pushed the od-fix-flacky-poll-tests branch from 0d087d5 to 054cb76 Compare June 15, 2024 13:32

lesterchoi-okta reviewed Jun 18, 2024

View reviewed changes

yannongli-okta reviewed Jun 20, 2024

View reviewed changes

denysoblohin-okta force-pushed the od-fix-flacky-poll-tests branch from 723f794 to ffa12d6 Compare June 27, 2024 15:57

Luxu2-Okta previously approved these changes Aug 9, 2024

View reviewed changes

Fix flaky TestCafe OV polling test in DeviceChallengePollView_spec

b197e89

denysoblohin-okta dismissed Luxu2-Okta’s stale review via b197e89 August 22, 2024 12:13

denysoblohin-okta force-pushed the od-fix-flacky-poll-tests branch from ffa12d6 to b197e89 Compare August 22, 2024 12:13

restore Promise.resolve()

568bd68

denysoblohin-okta changed the base branch from master to 7.21 August 22, 2024 12:30

need to reject

a5b414c

denysoblohin-okta force-pushed the od-fix-flacky-poll-tests branch from 1e31c67 to a5b414c Compare August 22, 2024 13:48

Luxu2-Okta approved these changes Aug 22, 2024

View reviewed changes

lesterchoi-okta approved these changes Aug 22, 2024

View reviewed changes

lesterchoi-okta requested changes Aug 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky TestCafe OV polling test in `DeviceChallengePollView_spec` #3654

Fix flaky TestCafe OV polling test in `DeviceChallengePollView_spec` #3654

denysoblohin-okta commented Jun 14, 2024

lesterchoi-okta Jun 18, 2024

lesterchoi-okta Jun 18, 2024

denysoblohin-okta Jun 18, 2024

shuowu-okta Jun 18, 2024

denysoblohin-okta Jun 19, 2024

yannongli-okta Jun 20, 2024

denysoblohin-okta Jun 21, 2024

lesterchoi-okta Jun 25, 2024

Luxu2-Okta Aug 7, 2024

lesterchoi-okta Aug 20, 2024

Luxu2-Okta Aug 20, 2024

denysoblohin-okta Aug 22, 2024 •

edited

Loading

denysoblohin-okta Aug 22, 2024

Luxu2-Okta Aug 22, 2024

shawyu-okta Aug 22, 2024

Luxu2-Okta left a comment

Luxu2-Okta left a comment

lesterchoi-okta left a comment •

edited

Loading

Fix flaky TestCafe OV polling test in DeviceChallengePollView_spec #3654

Are you sure you want to change the base?

Fix flaky TestCafe OV polling test in DeviceChallengePollView_spec #3654

Conversation

denysoblohin-okta commented Jun 14, 2024

Description:

PR Checklist

Issue:

Reviewers:

Screenshot/Video:

Downstream Monolith Build:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denysoblohin-okta Aug 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Luxu2-Okta left a comment

Choose a reason for hiding this comment

Luxu2-Okta left a comment

Choose a reason for hiding this comment

lesterchoi-okta left a comment • edited Loading

Choose a reason for hiding this comment

Fix flaky TestCafe OV polling test in `DeviceChallengePollView_spec` #3654

Fix flaky TestCafe OV polling test in `DeviceChallengePollView_spec` #3654

denysoblohin-okta Aug 22, 2024 •

edited

Loading

lesterchoi-okta left a comment •

edited

Loading