Fix Importing Test Results Failure for MsTest for Ignored Scenarios #378

nnanoob · 2016-09-30T07:05:28Z

fix null featureTreeNode in ApplyTestResultsToFeature
handle null value for GetExampleResults and defaulted to TestResult.Inconclusive

+ handle null value for GetExampleResults and defaulted to TestResult.Inconclusive

dirkrombauts · 2016-09-30T13:06:15Z

src/Pickles/Pickles/Runner.cs

            {
-                var featureTreeNode = node as FeatureNode;


This change has nothing to do with the fix, has it? Please do me a favour and resist giving in to ReSharper's style suggestions while doing a fix. I will be happy to review this kind of changes but please send them in a separate pull request.

this change is actually required because the original code, featureTreeNode = node as FeatureNode will returns null and no test results will be imported

You're right, I stand corrected.

Wouldn't it be simpler then to use the OfType extension method?

refactored to use OfType. Was copy pasting sample code from ImageNode earlier. But I didn't refactored those.

dirkrombauts · 2016-09-30T13:08:18Z

src/Pickles/Pickles.TestFrameworks/MsTest/MsTestSingleResults.cs

@@ -137,6 +137,10 @@ public override TestResult GetExampleResult(ScenarioOutline scenario, string[] e
            var scenarioElements = this.GetScenariosForScenarioOutline(scenario);

            var theScenario = this.GetScenarioThatMatchesTheExampleValues(scenario, exampleValues, scenarioElements);
+            if (theScenario == null)


Thanks for this fix. I'm following a test-first policy in Pickles, so please provide some tests for this case. It is worth checking out the other test result providers because the same problem may be lurking there.

See CONTRIBUTING.md for information on how to contribute code to the test result providers.

added test for MsTest under WhenParsingMsTestResultsFileWithIgnoredExample.cs

…xamples

dirkrombauts · 2016-09-30T16:41:31Z

src/Pickles/Pickles/Runner.cs

            {
-                var featureTreeNode = node as FeatureNode;


You're right, I stand corrected.

Wouldn't it be simpler then to use the OfType extension method?

dirkrombauts · 2016-09-30T16:43:43Z

...es/Pickles.TestFrameworks.UnitTests/MsTest/WhenParsingMsTestResultsFileWithIgnoredExample.cs

+    public class WhenParsingMsTestResultsFileWithIgnoredExample : StandardTestSuite<MsTestResults>
+    {
+        public WhenParsingMsTestResultsFileWithIgnoredExample()
+            : base("MsTest." + "results-example-mstest-ignoredexample.trx")


Simply for my understanding: why did you chose ignored examples. Is there anything magic about ignored in this case?

yes. The fix was in MSTest GetExampleResult which deal with ScenarioOutline

I know that the fix is about Scenario Outlines :-) What I wanted to know was why are you using an Ignored example? Would it also work if you have "normal" examples?

The original code will work fine if all are "normal" examples, but exception will be thrown if one of the examples are "ignored". A sample SpecFlow scenario below will results in exception

Scenario Outline: An example outline
Given I have an example
Examples:
| TestCase |
| 1 |

@ignore
Examples:
| TestCase |
| 2 |

What would happen if we use this scenario outline:

Given I have an example Examples: | TestCase | | 1 |

And then search for the example from row 2 - wouldn't that fail in just the same way in the original code?

sorry. kinda lost you here. But I'll try to answer :)
In the new code, the search on the row 2 (ignored) will still fail, but GetExampleResult method will handle it gracefully (check on null) and treat it as Inconclusive so that the whole cycle can be completed

I'll try to explain what I'm after in a different way.

I find the use of "ignored" here confusing. I think you can achieve the same test if you simply search for a example result that does not exist. Maybe I'm missing something. What value do you get from talking about ignored examples instead of normal examples?

Aha. I got you now. I'm just writing the test from the perspective of how a real life scenario will be (where you will be having an Example which is ignored and which resulted in the example results not to exists. The example results does not go missing by itself unless it was ignored or someone updated the .feature and then trying to generate the doc (with .trx that is ran with older .feature). I believe the first scenario is more likely to happen, thus that explain the test

I get it now. You're right, it is more likely that a result is not found because the scenario was ignored that that it is not found because the scenario was changed after the tests were run.

* Delete CNAME as advised by github support * Create CNAME * Get rid of ngenerics (#370) * Use Tree instead of NGenerics.GeneralTree * Use visitor instead of ActionVisitor * Rename Visitor to Traversor * Remove reference to unused NGenerics library * Fix whitespace * Remove NGenerics from nuspecs and build scripts * Get rid of traversor * Compatibility with nunit.console 3.4 and nunit.framework 2.6 (#369) * Compatibility with nunit.console 3.4 and nunit.framework 2.6 * Added example output file and test for nunit 2 tests ran with nunit 3 runner. * Add "NUnit 2 tests run by NUnit 3" to test harness (#371) * Move generation and commit of output to DeployArtifacts (#372) * Added test for sorting when generating word docs (#373) * Added test for sorting when generating word docs * Use stream overloads of WordprocessingDocument * Improve tree (#374) * Add test about sorting * Make it compile * Make it pass * Check for null in constructor * Add test about null names in iteration * Factory method for creating trees * Check for null tree when adding * Add test for null node * Release 2.8.3 (#377) * Release 2.6.3 (#347) * Enhancement to support Unix path! (#344) I'm not sure you already got a try or a feedback about it but actually pickles run relatively well under Linux with mono which is interesting when like me you can't get a windows machine. However without this little change we can't get the folder structure display correctly. Could you integrate this modification? Thanks * Version Bump * Update change log * Release 2.7.0 (#351) * Enhancement to support Unix path! (#344) I'm not sure you already got a try or a feedback about it but actually pickles run relatively well under Linux with mono which is interesting when like me you can't get a windows machine. However without this little change we can't get the folder structure display correctly. Could you integrate this modification? Thanks * Release 2.6.3 (#348) * Version Bump * Update change log * Add --enableComments Flag to Command Line - Default to True (#346) * Command-Line Flag --comments - Default to True - Incl Unit Test * Changed Command Line Argument --comments to --enableComments * Added Enable Comments Option to MSBuild - Powershell - WPF Runners * Fixed Bug with enableComments Property Not Being Used in MainViewModel * Add EnableComments to the targets file * Adapt change log * Version bump to 2.7.0 * Release 2.8.0 (#355) * Enhancement to support Unix path! (#344) I'm not sure you already got a try or a feedback about it but actually pickles run relatively well under Linux with mono which is interesting when like me you can't get a windows machine. However without this little change we can't get the folder structure display correctly. Could you integrate this modification? Thanks * Release 2.6.3 (#348) * Version Bump * Update change log * Add --enableComments Flag to Command Line - Default to True (#346) * Command-Line Flag --comments - Default to True - Incl Unit Test * Changed Command Line Argument --comments to --enableComments * Added Enable Comments Option to MSBuild - Powershell - WPF Runners * Fixed Bug with enableComments Property Not Being Used in MainViewModel * Release 2.7.0 (#352) * Release 2.6.3 (#347) * Enhancement to support Unix path! (#344) I'm not sure you already got a try or a feedback about it but actually pickles run relatively well under Linux with mono which is interesting when like me you can't get a windows machine. However without this little change we can't get the folder structure display correctly. Could you integrate this modification? Thanks * Version Bump * Update change log * Add EnableComments to the targets file * Adapt change log * Version bump to 2.7.0 * #320 - Scenario Deep Linking - DHTML and HTML Output (#350) * #320 - Scenario Deep Linking - DHTML and HTML Output * #320 - Fix Failing Unit Tests * #320 - Resolved Issue with Image Resource Not Copying * #320 - Add Backward-Compatibility for Hashed Feature Path * #320 - Added Modal Dialog for Link Copy * Edit release notes * Version Bump (2.8.0) * Release 2.8.1 (#360) * Add release notes * Fix some markdown syntax things * Bump version number (2.8.3) * Version 2.8.3 * Special chars need to be replaced (#375) * Special chars need to be replaced There are some more special chars (especially german Umlaute) that need to be replaced for the match to be successfull. * Special chars need to be replaced #375 Creating test cases and fixing handling of umlauts and ampersand * Changed indentation * Fix Importing Test Results failure for MsTest (#378) * + fix null featureTreeNode in ApplyTestResultsToFeature + handle null value for GetExampleResults and defaulted to TestResult.Inconclusive * + added extra test scenario for MsTest with ignored ScenarioOutline examples * + refactored LINQ for FeatureNode in ApplyTestResultsToFeatures * Show parser failures and quit with an error (#379) * Don't swallow exceptions if parsing a feature file throws. If parsing fails we want the build to break. We don't want to silently ignore parts. * Improve NLog layout for console - Don't show datetime, loglevel and logger - Use colors - Show exception messages * Version Bump (2.9.0)

dirkrombauts · 2016-10-07T15:18:11Z

Released in version 2.9.0.

+ fix null featureTreeNode in ApplyTestResultsToFeature

b3633b3

+ handle null value for GetExampleResults and defaulted to TestResult.Inconclusive

dirkrombauts requested changes Sep 30, 2016

View reviewed changes

+ added extra test scenario for MsTest with ignored ScenarioOutline e…

6ca2573

…xamples

dirkrombauts reviewed Sep 30, 2016

View reviewed changes

+ refactored LINQ for FeatureNode in ApplyTestResultsToFeatures

846d448

dirkrombauts approved these changes Oct 5, 2016

View reviewed changes

dirkrombauts merged commit 3115b2e into picklesdoc:develop Oct 5, 2016

dirkrombauts added the ready-to-be-released label Oct 5, 2016

dirkrombauts changed the title ~~Fix Importing Test Results failure for MsTest~~ Fix Importing Test Results Failure for MsTest for Ignored Scenarios Oct 7, 2016

This was referenced Oct 7, 2016

Release 2.9.0 #380

Merged

Release 2.9.0 #381

Merged

dirkrombauts removed the ready-to-be-released label Oct 7, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Importing Test Results Failure for MsTest for Ignored Scenarios #378

Fix Importing Test Results Failure for MsTest for Ignored Scenarios #378

nnanoob commented Sep 30, 2016

dirkrombauts Sep 30, 2016

nnanoob Sep 30, 2016

dirkrombauts Sep 30, 2016 •

edited

Loading

nnanoob Oct 4, 2016

dirkrombauts Sep 30, 2016

nnanoob Sep 30, 2016

dirkrombauts Sep 30, 2016 •

edited

Loading

dirkrombauts Sep 30, 2016

nnanoob Oct 4, 2016

dirkrombauts Oct 4, 2016

nnanoob Oct 4, 2016

dirkrombauts Oct 4, 2016

nnanoob Oct 5, 2016

dirkrombauts Oct 5, 2016

nnanoob Oct 5, 2016

dirkrombauts Oct 5, 2016

dirkrombauts commented Oct 7, 2016

Fix Importing Test Results Failure for MsTest for Ignored Scenarios #378

Fix Importing Test Results Failure for MsTest for Ignored Scenarios #378

Conversation

nnanoob commented Sep 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dirkrombauts Sep 30, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dirkrombauts Sep 30, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dirkrombauts commented Oct 7, 2016

dirkrombauts Sep 30, 2016 •

edited

Loading

dirkrombauts Sep 30, 2016 •

edited

Loading