LivingDoc Implementation (Fix for skipped steps and missing error messages)

[Registeel]

I have taken an implementation of @dakiers project that generates the TestExecution.json file that is required to run SpecFlow+ LivingDoc with test results. I added everything but the "Hooks" directory to a class library then added the hooks to my existing reqnroll project and it worked great for generating the TestExecution.json file.

There are issues I found that I have resolved in this implementation.

1. LivingDoc no longer records the skipped steps as grayed out and shows them as successful.
2. step.TestError?.Message is always empty so errored steps give no output.

I've resolved both of these issues. I'll start with the first problem.

Skipped Steps

The issue with skipped steps is that they are not processed in the [AfterStep] hook. So once you hit a failed step, all the skipped ones afterwards are not recorded in the ExecutionResult object that is written to in the [AfterStep] hook like so:

_result!.StepResult.Add(new(
    (DateTime.UtcNow - _stepStartTime!).Value, 
    step.Status, 
    step.TestError?.Message, 
    outputHelper.FlushOutputs(OutputLocation.AfterStep))); 

The solution for this problem is to build a custom SkippedStepHandler. This is a custom class with any name you like that inherits the reqnroll ISkippedStepHandler interface.

My implementation looks like this:

public class SkippedStepHandler : ISkippedStepHandler
{
    public void Handle(ScenarioContext scenarioContext)
    {
        var executionResult = scenarioContext.ScenarioContainer.Resolve<ExecutionResult>();

        executionResult!.StepResults.Add(new(
            new TimeSpan(),
            scenarioContext.StepContext.Status,
            scenarioContext.StepContext.TestError?.Message));
    }
}

The confusing part about this is where the ExecutionResult is coming from. This is what is used to record the step status to write to the TestExecution.json file. We need to pass this object by reference from the main reqnroll hook file that is running the tests.

I have done this in my [BeforeScenario] hook from dakiers project like so:

[BeforeScenario]
public void BeforeScenario(FeatureInfo feature, ScenarioInfo scenario)
{
    var skipHandler = new SkippedStepHandler();
    this._objectContainer.RegisterInstanceAs<ISkippedStepHandler>(skipHandler);

    _result = new(
        ExecutionContextType,
        feature.FolderPath,
        feature.Title,
        feature.Title,
        scenario.Arguments.Values.Cast<string>().ToArray());

    this._objectContainer.RegisterInstanceAs(_result);
}

The _objectContainer is an instance of an IObjectContainer that is dependency injected into the main Hook class.

With this setup the _result object is passed to the custom handler and when skipped steps are hit they are written to the result before being written to the TestExecution.json file in the [AfterScenario] and [AfterTestRun] hooks.

Onto problem number two.

Empty Error Messages

This one was a simpler problem. Essentially step.TestError?.Message is always empty, but you do get an error message for the failure of the step in your Scenario Context.

My solution is to change step.TestError?.Message, (step coming from var step = scenario.StepContext), to scenario.TestError?.Message.
If you would like extra logging I also appended scenario.TestError?.InnerException?.Message to mine.

[gasparnagy]

@Registeel Thx for sharing!