[e2e] Adding failure details for driver tests #2593

nturgut · 2020-03-10T00:51:48Z

Description

Adding failure details for driver tests.

Before the change the failure output was something like:

[+2616 ms] result fail
[  +88 ms] Stopping application instance.
[        ] Stopping application.
[   +2 ms] "flutter drive" took 22,861ms.

The updated output:

[+2782 ms] Failure Details:
[        ] Failure in method: verify text
[        ] ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
[   +1 ms] The following TestFailure object was thrown running a test:
[        ]   Expected: <3>
[        ]   Actual: <2>
[        ] When the exception was thrown, this was the stack:
[        ]     at Object.wrapException (http://localhost:36633/main.dart.js:3646:17)
[        ]     at Object.throwExpression (http://localhost:36633/main.dart.js:3659:15)
[        ]     at Object.fail (http://localhost:36633/main.dart.js:16457:16)
[        ]     at Object._expect (http://localhost:36633/main.dart.js:16454:9)
[        ]     at Object.expect0 (http://localhost:36633/main.dart.js:16430:9)
[        ]     at Object.expect (http://localhost:36633/main.dart.js:19782:9)
[        ]     at http://localhost:36633/main.dart.js:82397:17
[        ]     at _wrapJsFunctionForAsync_closure.$protected (http://localhost:36633/main.dart.js:6615:15)
[        ]     at _wrapJsFunctionForAsync_closure.call$2 (http://localhost:36633/main.dart.js:37105:12)
[        ]     at StackZoneSpecification__registerBinaryCallback__closure.call$0 (http://localhost:36633/main.dart.js:80439:21)
[        ] The test description was:
[        ]   verify text
[        ] ═════════════════════════════════════════════════════════════════
[        ] end of failure 1
[        ] Failure in method: another failure
[        ] ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
[        ] The following TestFailure object was thrown running a test:
[        ]   Expected: <3>
[        ]   Actual: <2>
[        ] When the exception was thrown, this was the stack:
[        ]     at Object.wrapException (http://localhost:36633/main.dart.js:3646:17)
[        ]     at Object.throwExpression (http://localhost:36633/main.dart.js:3659:15)
[        ]     at Object.fail (http://localhost:36633/main.dart.js:16457:16)
[        ]     at Object._expect (http://localhost:36633/main.dart.js:16454:9)
[        ]     at Object.expect0 (http://localhost:36633/main.dart.js:16430:9)
[        ]     at Object.expect (http://localhost:36633/main.dart.js:19782:9)
[        ]     at http://localhost:36633/main.dart.js:82426:17
[        ]     at _wrapJsFunctionForAsync_closure.$protected (http://localhost:36633/main.dart.js:6615:15)
[        ]     at _wrapJsFunctionForAsync_closure.call$2 (http://localhost:36633/main.dart.js:37105:12)
[        ]     at StackZoneSpecification__registerBinaryCallback__closure.call$0 (http://localhost:36633/main.dart.js:80439:21)
[        ] The test description was:
[        ]   another failure
[        ] ═════════════════════════════════════════════════════════════════
[        ] end of failure 2
[  +71 ms] Stopping application instance.
[        ] Stopping application.
[   +2 ms] "flutter drive" took 23,618ms.

Related Issues

Fixes: flutter/flutter#51940

Checklist

Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

Breaking Change

Does your PR require plugin users to manually update their apps to accommodate your change?

Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
No, this is not a breaking change.

packages/e2e/README.md

packages/e2e/example/test_driver/example_e2e_test.dart

packages/e2e/lib/common.dart

yjbanov · 2020-03-11T18:14:20Z

packages/e2e/lib/common.dart

+  String get failureDetails => _allTestsPassed ? '' : _failureDetails;
+
+  /// Convert a string pass/fail result to a boolean.
+  static bool allTestsPassed(String result) => (result == 'pass')


Does this need to be public?

I am providing as many public methods as possible.

I think you already noticed. This is a breaking change. There are many tests already written using e2e. I am trying to provide methods to developers:

a method where they can call if they want to keep their string usage ('pass', 'fail')

a method where they can call for a boolean.

yjbanov · 2020-03-11T18:15:51Z

packages/e2e/lib/common.dart

+}
+
+/// Method for formating the test failures' details.
+String formatFailures(Map<String, String> failureDetails) {


If failure details are a Map<String, String> then it can be turned to JSON as is. Why format? Formatting loses information that could be used on the receiving end.

Cause formatting looks better on the output. It adds extra empty lines for example.

For me it increased my productivity since I am able to see things a lot more clear. You don't have to use it in your tests though. This one is not a mandatory method.

For giving more context this is as json:

{"verify text":"══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════\nThe following TestFailure object was thrown running a test:\n Expected: <5>\n Actual: <4>\n\nWhen the exception was thrown, this was the stack:\n at Object.wrapException (http://localhost:8080/main.dart.js:3646:17)\n at Object.throwExpression (http://localhost:8080/main.dart.js:3659:15)\n at Object.fail (http://localhost:8080/main.dart.js:16444:16)\n at Object._expect (http://localhost:8080/main.dart.js:16441:9)\n at Object.expect0 (http://localhost:8080/main.dart.js:16417:9)\n at Object.expect (http://localhost:8080/main.dart.js:19769:9)\n at http://localhost:8080/main.dart.js:82327:17\n at _wrapJsFunctionForAsync_closure.$protected (http://localhost:8080/main.dart.js:6615:15)\n at _wrapJsFunctionForAsync_closure.call$2 (http://localhost:8080/main.dart.js:37092:12)\n at StackZoneSpecification__registerBinaryCallback__closure.call$0 (http://localhost:8080/main.dart.js:80369:21)\n\nThe test description was:\n verify text\n═════════════════════════════════════════════════════════════════\n","fail text":"══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════\nThe following TestFailure object was thrown running a test:\n Expected: <5>\n Actual: <4>\n\nWhen the exception was thrown, this was the stack:\n at Object.wrapException (http://localhost:8080/main.dart.js:3646:17)\n at Object.throwExpression (http://localhost:8080/main.dart.js:3659:15)\n at Object.fail (http://localhost:8080/main.dart.js:16444:16)\n at Object._expect (http://localhost:8080/main.dart.js:16441:9)\n at Object.expect0 (http://localhost:8080/main.dart.js:16417:9)\n at Object.expect (http://localhost:8080/main.dart.js:19769:9)\n at http://localhost:8080/main.dart.js:82349:17\n at _wrapJsFunctionForAsync_closure.$protected (http://localhost:8080/main.dart.js:6615:15)\n at _wrapJsFunctionForAsync_closure.call$2 (http://localhost:8080/main.dart.js:37092:12)\n at StackZoneSpecification__registerBinaryCallback__closure.call$0 (http://localhost:8080/main.dart.js:80369:21)\n\nThe test description was:\n fail text\n═════════════════════════════════════════════════════════════════\n"}

this is with my format method:

Failure Details: Failure in method: verify text ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════ The following TestFailure object was thrown running a test: Expected: <5> Actual: <4> When the exception was thrown, this was the stack: at Object.wrapException (http://localhost:8080/main.dart.js:3646:17) at Object.throwExpression (http://localhost:8080/main.dart.js:3659:15) at Object.fail (http://localhost:8080/main.dart.js:16453:16) at Object._expect (http://localhost:8080/main.dart.js:16450:9) at Object.expect0 (http://localhost:8080/main.dart.js:16426:9) at Object.expect (http://localhost:8080/main.dart.js:19778:9) at http://localhost:8080/main.dart.js:82348:17 at _wrapJsFunctionForAsync_closure.$protected (http://localhost:8080/main.dart.js:6615:15) at _wrapJsFunctionForAsync_closure.call$2 (http://localhost:8080/main.dart.js:37101:12) at StackZoneSpecification__registerBinaryCallback__closure.call$0 (http://localhost:8080/main.dart.js:80390:21) The test description was: verify text ═════════════════════════════════════════════════════════════════ end of failure 1 Failure in method: fail text ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════ The following TestFailure object was thrown running a test: Expected: <5> Actual: <4> When the exception was thrown, this was the stack: at Object.wrapException (http://localhost:8080/main.dart.js:3646:17) at Object.throwExpression (http://localhost:8080/main.dart.js:3659:15) at Object.fail (http://localhost:8080/main.dart.js:16453:16) at Object._expect (http://localhost:8080/main.dart.js:16450:9) at Object.expect0 (http://localhost:8080/main.dart.js:16426:9) at Object.expect (http://localhost:8080/main.dart.js:19778:9) at http://localhost:8080/main.dart.js:82370:17 at _wrapJsFunctionForAsync_closure.$protected (http://localhost:8080/main.dart.js:6615:15) at _wrapJsFunctionForAsync_closure.call$2 (http://localhost:8080/main.dart.js:37101:12) at StackZoneSpecification__registerBinaryCallback__closure.call$0 (http://localhost:8080/main.dart.js:80390:21) The test description was: fail text ═════════════════════════════════════════════════════════════════ end of failure 2

I think I prefer to use the second one. It is more readable hence increases productivity.

Ok, I stopped using the formatted result. I added a method which developers can use for a formatted output though.

I also added it to our tests:

final common.Response response = common.Response.fromJson(jsonResult); await driver.close(); if (response.allTestsPassed) { print('All tests passed.'); exit(0); } else { response.formattedFailureDetails; print('Failure Details:\n${response.formattedFailureDetails}'); exit(1); }

packages/e2e/lib/e2e.dart

packages/e2e/pubspec.yaml

collinjackson

Could the failures be returned as a List? It seems like there's some useful ordering info that will get lost if we do a Map.

packages/e2e/example/test_driver/example_e2e_test.dart

packages/e2e/README.md

packages/e2e/example/test_driver/example_e2e_test.dart

…o e2e package

nturgut · 2020-03-25T17:09:35Z

Could the failures be returned as a List? It seems like there's some useful ordering info that will get lost if we do a Map.

I just realized I missed this one. I'll have a look now :)

nturgut · 2020-03-25T22:37:32Z

Could the failures be returned as a List? It seems like there's some useful ordering info that will get lost if we do a Map.

I just realized I missed this one. I'll have a look now :)

I changed the type to List.

collinjackson

lgtm with one minor nit

collinjackson · 2020-03-26T01:04:09Z

packages/e2e/lib/e2e_driver.dart

+import 'dart:async';
+import 'dart:io';
+
+import 'package:e2e/common.dart' as e2ecommon;


I would stick to either e2e or common or just don't namespace the import, e2ecommon looks unusual to me.

Thanks for the review!

I'll use e2e then. 'Response' collides with another name. Hence I had to use a namespace.

nturgut · 2020-03-27T00:28:44Z

Merging the PR. Looks like tests are passing.

* Adding failure details for driver tests * formate fix * addressing reviewer comments * addressing the reviewer comments. part 2 * fix error in response_data * addressing reviewer comments * fixing formatting * addressin reviewer comments. Carrying the repeated driver test code to e2e package * chaning the failure's details type from map to list * format changes * add documentation to public members * formatting * changing namespace name

nturgut requested a review from collinjackson March 10, 2020 00:51

nturgut requested a review from digiter as a code owner March 10, 2020 00:51

googlebot added the cla: yes label Mar 10, 2020

nturgut requested review from ditman and yjbanov and removed request for digiter and ditman March 10, 2020 00:54

yjbanov reviewed Mar 11, 2020

View reviewed changes

collinjackson self-assigned this Mar 12, 2020

Nurhan Turgut added 4 commits March 13, 2020 16:37

Adding failure details for driver tests

d13848e

formate fix

a9e64fc

addressing reviewer comments

13fe443

addressing the reviewer comments. part 2

861d8d2

nturgut force-pushed the error_message_update branch from 2dd4da4 to 861d8d2 Compare March 13, 2020 23:38

nturgut requested a review from yjbanov March 13, 2020 23:38

fix error in response_data

bb0d2e0

yjbanov approved these changes Mar 16, 2020

View reviewed changes

yjbanov reviewed Mar 16, 2020

View reviewed changes

packages/e2e/pubspec.yaml Outdated Show resolved Hide resolved

nturgut added 2 commits March 20, 2020 19:12

addressing reviewer comments

e3994de

fixing formatting

662a81c

collinjackson reviewed Mar 24, 2020

View reviewed changes

addressin reviewer comments. Carrying the repeated driver test code t…

f39e578

…o e2e package

chaning the failure's details type from map to list

ec3a49b

nturgut added 3 commits March 25, 2020 15:39

format changes

e086c61

add documentation to public members

74f5bef

formatting

e57f818

collinjackson approved these changes Mar 26, 2020

View reviewed changes

changing namespace name

405372e

nturgut merged commit bce60e6 into flutter:master Mar 27, 2020

nturgut mentioned this pull request Oct 28, 2020

flutter drive does not output print, swallows exception info when test fails in profile/release modes flutter/flutter#69070

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[e2e] Adding failure details for driver tests #2593

[e2e] Adding failure details for driver tests #2593

nturgut commented Mar 10, 2020 •

edited

Loading

yjbanov Mar 11, 2020

nturgut Mar 13, 2020

yjbanov Mar 11, 2020

nturgut Mar 13, 2020

nturgut Mar 13, 2020 •

edited

Loading

nturgut Mar 21, 2020

nturgut Mar 21, 2020

collinjackson left a comment

nturgut commented Mar 25, 2020

nturgut commented Mar 25, 2020

collinjackson left a comment

collinjackson Mar 26, 2020

nturgut Mar 26, 2020

nturgut commented Mar 27, 2020

[e2e] Adding failure details for driver tests #2593

[e2e] Adding failure details for driver tests #2593

Conversation

nturgut commented Mar 10, 2020 • edited Loading

Description

Related Issues

Checklist

Breaking Change

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nturgut Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collinjackson left a comment

Choose a reason for hiding this comment

nturgut commented Mar 25, 2020

nturgut commented Mar 25, 2020

collinjackson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nturgut commented Mar 27, 2020

nturgut commented Mar 10, 2020 •

edited

Loading

nturgut Mar 13, 2020 •

edited

Loading