Polymer tests are flaky in Dartium

[vsmenon]

We're not getting much useful tracking on these since the 35 roll. Not seeing anything similar outside these polymer tests.

John / Siggi: any ideas what might be different here? Have you seen anything similar in dart2js?

polymer/example/component/news/test/news_index_test: Pass, RuntimeError # Issue #8931
polymer/test/attr_deserialize_test: Pass, RuntimeError # Issue #18931
polymer/test/attr_mustache_test: Pass, RuntimeError # Issue #18931
polymer/test/bind_test: Pass, RuntimeError # Issue #18931
polymer/test/custom_event_test: Pass, RuntimeError # Issue #18931
polymer/test/entered_view_test: Pass, RuntimeError # Issue #18931
polymer/test/event_handlers_test: Pass, RuntimeError # Issue #18931
polymer/test/event_path_test: Pass, RuntimeError # Issue #18931
polymer/test/event_path_declarative_test: Pass, RuntimeError, Timeout # Issue #18931
polymer/test/events_test: Pass, RuntimeError # Issue #18931
polymer/test/instance_attrs_test: Pass, RuntimeError # Issue #18931
polymer/test/js_interop_test: Pass, RuntimeError # Issue #18931
polymer/test/nested_binding_test: Pass, RuntimeError # Issue #18931
polymer/test/noscript_test: Pass, RuntimeError # Issue #18931
polymer/test/prop_attr_bind_reflection_test: Pass, RuntimeError # Issue #18931
polymer/test/prop_attr_reflection_test: Pass, RuntimeError # Issue #18931
polymer/test/property_change_test: Pass, Timeout # 8931
polymer/test/publish_attributes_test: Pass, RuntimeError # Issue #18931
polymer/test/publish_inherited_properties_test: Pass, RuntimeError # Issue #18931
polymer/test/register_test: Pass, RuntimeError # Issue #18931
polymer/test/take_attributes_test: Pass, RuntimeError # Issue #18931
polymer/test/template_distribute_dynamic_test: Pass, RuntimeError # Issue #18931

[kasperl]

I assume this is something you want to try to fix for 1.5?

---

Removed Priority-Unassigned label.
Added Priority-High label.

[sigmundch]

We have not been able to reproduce those flakes entirely.

They definitely don't fail in dart2js. We've done some things that we believe will help make it less likely to happen (I removed the use of the boot.js that we had at the time of the roll, and we are upgrading to the latest version of the polyfills).

I chatted with Rico about getting flaky history information, and unfortunately we don't have flaky logs for the Dartium bots, so he mentioned that the best we can do is remove the status lines and see what happens.

John plans to submit https://codereview.chromium.org/307793002 today, which will remove these pass/fail status lines.

[vsmenon]

John tried to remove but had to revert.

Let's try again after I roll to 36.0.1985.49 (https://code.google.com/p/dart/issues/detail?id=19269).

There are a few stability fixes there, include a couple related to html imports:
http://build.chromium.org/f/chromium/perf/dashboard/ui/changelog_blink.html?url=/branches/chromium/1985&range=173811:175393&mode=html

Do you think these might be related? I suspect our Polymer tests are the only ones using imports extensively.

[kasperl]

Adding Vijay as owner of this (under the assumption that the next step is to try to bring in the Blink stability fixes).

---

Set owner to @vsmenon.

[jmesserly]

ah yes, that could totally be the issue. Thank you Vijay! :D

[kasperl]

Any update on this?

[leafpetersen]

I'm looking into this, I can reproduce locally. I'll update later today.

[vsmenon]

We've replicated the underlying issue (custom elements not upgrading at the expected time) with upstream Chrome (36 beta and 37 canary).

It only appears to occur if two frames are concurrently loading pages with custom elements - hence why we're seeing it in the test framework.

Dropping from this milestone. We still need a way to re-enable polymer tests - perhaps working around this bug if necessary.

---

Removed this from the 1.5 milestone.
Added this to the 1.6 milestone.

[sigmundch]

because we don't use iframes with content_shell, we have been able to run polymer tests there. I can't recall exactly when we did the switch in the status files, but we have already enabled the tests in content_shell in bleeding_edge.

[vsmenon]

Removed this from the 1.6 milestone.
Added this to the 1.7 milestone.

[vsmenon]

Removed this from the 1.7 milestone.
Added this to the 1.8 milestone.

[whesse]

This test is flaky on dartium now:
polymer/example/component/news/test/news_index_test

It sounds like the same problem - the assertion that fails is
 expect(listComp is Polymer, true, reason: 'x-news should be created');

All the other tests in the list in the top comment remain marked as flaky in the status file, but this test was removed - it was marked failing before polymer 0.4, and removed in https://codereview.chromium.org//558673004. Restoring its flaky status.

[vsmenon]

Removed this from the 1.8 milestone.

[kasperl]

Added this to the 1.9 milestone.

[kasperl]

Added Accepted label.

[vsmenon]

Removed this from the 1.9 milestone.
Added this to the 1.10 milestone.

[vsmenon]

Is this still an issue?

---

Removed the owner.
Added Triaged label.

[sigmundch]

It might be - we need to investigate. The flaky status is still in our .status file in polymer-dart here: https://github.com/dart-lang/polymer-dart/blob/master/.status

There was a bug creating the .flaky logs in the package-bots that was fixed this week, so we should be able to take another look.

[sigmundch]

cc @jakemac53.