What looks like passing test is reported as failure

[peter-ahe-google]

From http://build.chromium.org/p/client.dart/builders/dartium-win-inc-be/builds/891/steps/drt_core_unchecked_tests/logs/stdio:

FAILED: none-drt release_ia32 html/svgelement_test/supported_animateTransform
Expected: Pass
Actual: RuntimeError
CommandOutput[content_shell]:

stdout:
Content-Type: text/plain
PASS
8 PASS Expectation: supported_animateTransform supported.
All 1 tests passed

EOF

EOF

stderr:

EOF

To retest, run: E:\b\build\slave\dartium-win-inc-be\build\src\dart\tools\testing\bin\windows\dart.exe /E:/b/build/slave/dartium-win-inc-be/build/src/dart/tools/testing/dart/http_server.dart -p 49484 -c 49485 --build-directory=E:\b\build\slave\dartium-win-inc-be\build\src\build\Release --runtime=drt

Command[content_shell]: E:\b\build\slave\dartium-win-inc-be\build\src\build\Release\content_shell.exe --no-timeout --dump-render-tree http://127.0.0.1:49484/root_build/generated_tests/none-drt/tests_html_svgelement_test/test.html?crossOriginPort=49485&group=supported_animateTransform
Took 0:00:00.702000

Short reproduction command (experimental):
    python tools/test.py -rdrt -mrelease -t240 html/svgelement_test/supported_animateTransform

[mkustermann]

This is a known issue. If you look closely enough, you'll see this:

2013-11-18 09:13:37.411 Warning: All tests passed, but exitCode != 0. (E:\b\build\slave\dartium-win-inc-be\build\src\build\Release\content_shell.exe --no-timeout --dump-render-tree http://127.0.0.1:49484/root_build/generated_tests/none-drt/tests_html_svgelement_test/test.html?crossOriginPort=49485&group=supported_animateTransform)

This happens very often and it only happens on content shell. Our current best guess is that the renderer crashes when shutting down.

---

cc @vsmenon.
cc @a-siva.
cc @ricowind.
Removed Area-Test label.
Added Area-Dartium label.

[peter-ahe-google]

I've filed:

Issue #15141 to track the problem that the test runner doesn't print the exitCode.

Issue #15142 to track the problem that the test runner doesn't include the above warning in the test summary.

[ricowind]

Vijay/Siva, is the underlying issue something we are looking into? This is causing a lot of flake on our bots, it seems that we are hitting this a lot on debug windows, did you guys try reproducing this on windows?
This could be related to
https://code.google.com/p/chromium/issues/detail?id=192438
based on the exitcode

---

Set owner to @vsmenon.
Removed Priority-Unassigned label.
Added Priority-High label.

[vsmenon]

I've been looking into this. Some quick notes:

• The error code is -1073740022. See here (and thanks for adding the logging!):

http://chromegw.corp.google.com/i/client.dart/builders/dartium-win-inc-be/builds/981/steps/drt_core_unchecked_tests/logs/stdio

• The content shell main appears to expect positive exit codes (and only short circuits on those):

http://src.chromium.org/viewvc/multivm/branches/1650/chrome/content/shell/browser/shell_browser_main.cc?revision=1520

• Chrome error codes should be positive:

http://src.chromium.org/viewvc/multivm/branches/1650/chrome/content/public/common/result_codes_list.h?revision=1520

Can we check for a positive / valid exit code in our test scripts?

[ricowind]

• The error code is -1073740022. See here (and thanks for adding the logging!):
  Hence my reference to
  https://code.google.com/p/chromium/issues/detail?id=192438
  -1073740022 is 0xC000070A which that chromium bug refers to

[peter-ahe-google]

I've filed issue #15261 to track the problem that test runner prints confusing exit codes on Windows.

[ricowind]

Issue #7536 has been merged into this issue.

[vsmenon]

In the content shell main, the exit code is a signed 32-bit int. So, it actually is negative already in C++, not our scripts.

The content shell main does not treat a negative exit code as an error.

[ricowind]

Vijay: The assumption here is that this may very well be windows assigned error codes (that is the reason for the proposal for using err.exe on the exit code by Peter). If the main process is simply crashing - translating this error code by using the windows tool may give useful feedback

[kasperl]

Added this to the 1.6 milestone.

[vsmenon]

Removed this from the 1.6 milestone.
Added this to the 1.7 milestone.

[vsmenon]

Removed this from the 1.7 milestone.
Added this to the 1.8 milestone.

[vsmenon]

Removed this from the 1.8 milestone.
Added this to the 1.9 milestone.

[kasperl]

Added Accepted label.

[terrylucas]

supported_animateTransform is passing in roll 39 Dart release 1.9 on Windows.

---

Added CannotReproduce label.