How to manually run a scenario step by step?

[WBMPH]

Hi,

I'm trying to write a plugin or an app to manually start a scenario and step through each step one by one (kinda like a debugger). The reason I would want to load the scenario into my existing app is to re-use services without having to provide mocks for them. We have a physical robot and machines that integrate with this app over an OPCUA interface. For some actions, an operator interaction is required (eg Feed in item, take out processed item).

I was thinking if I would be able to do something like this:

• Compile the scenarios into a DLL
• Load the DLL into my app
• Parse all scenarios / steps
• Register the services
• Start a scenario
• Continue to next step

Another reason for this integration is for training an operator (eg we can setup an scenario and step through to train the operator how to handle it).

I found the plugin documentation however I think its quite bare and not really clear outside of some small config / DI to have any other effect on the test execution.

I was wondering how to hook into RuntimePluginTestExecutionLifecycleEvents.cs (BeforeStep & AfterStep)

[gasparnagy]

This is an interesting scenario and it is definitely possible, but not so easy to explain all the details.

Maybe you can achieve this with a plugin, but plugins can influence the behavior and they don't give you real control on how the test is executed. In case of Reqnroll, the tests are executed by the test runners (MsTest, NUnit, xUnit), so basically if you want to embed a Reqnroll test execution inside your app, you would need to do the same things what they do. (BTW, you can check the verify plugin to see an example of using the RuntimePluginTestExecutionLifecycleEvents, but I'm not sure this will be of your help.)

So instead of a plugin, I would recommend the following approach.

In Reqnroll the ITestRunner interface coordinates the execution of a test. It has methods, like OnScenarioStartAsync to indicate that you start a test and methods like GivenAsync or WhenAsync to instruct it to perform a step. (The step data you can get by using the Gherkin parser to parse your feature files.) Using the ITestRunner is not that easy though. What I can suggest is to have a look at the "code-behind" file generated for a feature file with a scenario and follow the same steps to acquire and prepare the ITestRunner to perform your steps. Make a small, simple prototype first.

[WBMPH]

Thanks for the feedback and I've come to your conclusion too that its going to be tricky but possible.

I've been able to extract all available steps and scenario classes through reflection however I also need to extract the location, title and description. That information is unfortunately not directly accessible through reflection without running the test as its generated in FeatureSetupAsync like this:

[NUnit.Framework.OneTimeSetUpAttribute()]
        public virtual async System.Threading.Tasks.Task FeatureSetupAsync()
        {
            testRunner = global::Reqnroll.TestRunnerManager.GetTestRunnerForAssembly(null, NUnit.Framework.TestContext.CurrentContext.WorkerId);
            global::Reqnroll.FeatureInfo featureInfo = new global::Reqnroll.FeatureInfo(new System.Globalization.CultureInfo("en-US"), "Scenarios", "SingleProduct", "We are creating a single product.", global::Reqnroll.ProgrammingLanguage.CSharp, featureTags);
            await testRunner.OnFeatureStartAsync(featureInfo);
        }

So I figured, I could just extend the NUnit Generator and add a custom attribute to the method and then extract that information from there. How hard can it be? (Famous last words). (Other option would be trying to parse the IL but yeah lets avoid that :))

So I created a plugin according to the docs like the following and I'm hoping I can omit some stuff by directly extending NUnit3TestGeneratorProvider
like this as I really dislike duplicating things:

[assembly: GeneratorPlugin(typeof(CodeGeneratorPlugin))]

namespace Simulation.Generator.ReqnrollPlugin;

public class CodeGeneratorPlugin : IGeneratorPlugin
{
    public void Initialize(GeneratorPluginEvents generatorPluginEvents, GeneratorPluginParameters generatorPluginParameters,
        UnitTestProviderConfiguration unitTestProviderConfiguration)
    {
        unitTestProviderConfiguration.UseUnitTestProvider("mphunit"); // My unique identifier for our generator
        generatorPluginEvents.CustomizeDependencies += OnCustomizeDependencies;
    }

    private void OnCustomizeDependencies(object? sender, CustomizeDependenciesEventArgs e)
    {
        e.ObjectContainer.RegisterTypeAs<CustomUnitTestGeneratorProvider, IUnitTestGeneratorProvider>("mphunit");
    }
}

and:

public class CustomUnitTestGeneratorProvider : NUnit3TestGeneratorProvider
{
    protected internal const string INFO_ATTRIBUTE = "Simulation.Shared.ScenarioInformationAttribute";
    public CustomUnitTestGeneratorProvider(CodeDomHelper codeDomHelper) : base(codeDomHelper)
    {
    }

    public override void SetTestClassInitializeMethod(TestClassGenerationContext generationContext)
    {
        base.SetTestClassInitializeMethod(generationContext);
        CodeDomHelper.AddAttribute(generationContext.TestClassInitializeMethod, INFO_ATTRIBUTE, generationContext.Feature.Location, generationContext.Feature.Name, generationContext.Feature.Description);
    }
}

I saw that the out of the box generators are registered in Reqnroll.Generator/UnitTestProvider/UnitTestGeneratorProviders.cs so having to register my new service according to the docs.

So after building the plugin I placed it in the folder containing my bdd tests .csproj file. When I now build, I receive the following exception:

Stacktrace

The "GenerateFeatureFileCodeBehindTask" task failed unexpectedly. System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. File name: 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type) at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext) at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments) at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg) at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType, Boolean mustBeInheritable, IList derivedAttributes, Boolean isDecoratedTargetSecurityTransparent) at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeAssembly assembly, RuntimeType caType) at System.Attribute.GetCustomAttributes(Assembly element, Type attributeType, Boolean inherit) at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) at Reqnroll.Generator.Plugins.GeneratorPluginLoader.LoadPlugin(PluginDescriptor pluginDescriptor) at Reqnroll.Generator.GeneratorContainerBuilder.LoadPlugins(ObjectContainer container, GeneratorPluginEvents generatorPluginEvents, UnitTestProviderConfiguration unitTestProviderConfiguration, IEnumerable`1 generatorPlugins) at Reqnroll.Generator.GeneratorContainerBuilder.CreateContainer(ReqnrollConfigurationHolder configurationHolder, ProjectSettings projectSettings, IEnumerable`1 generatorPluginInfos, IObjectContainer parentObjectContainer) at Reqnroll.Tools.MsBuild.Generation.WrappedGeneratorContainerBuilder.BuildGeneratorContainer(ReqnrollConfigurationHolder reqnrollConfigurationHolder, ProjectSettings projectSettings, IReadOnlyCollection`1 generatorPluginInfos, IObjectContainer rootObjectContainer) at Reqnroll.Tools.MsBuild.Generation.GenerateFeatureFileCodeBehindTaskExecutor.Execute() at Reqnroll.Tools.MsBuild.Generation.GenerateFeatureFileCodeBehindTask.Execute() at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() at Microsoft.Build.BackEnd.TaskBuilder.d__26.MoveNext()

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

So that is where I'm stuck with trying to extract data. In the meantime I'm working on getting the ITestRunner to work though've been busy so no major progress there yet for me.

[WBMPH]

Okay I got the code generator plugin to work. Three take aways:

1. I misread the docs about targeting netstandard2.0. I thought that was just for the runtime plugin. This resolved the assemblies not being found.

2. I removed the call unitTestProviderConfiguration.UseUnitTestProvider("mphunit"); as it would throw an error saying I've already picked a test runner.

3. I changed the registration from CustomizeDependencies to RegisterDependencies. My assumption is that this overrides the nunits' registration however I am not sure if there is some guaranteed order (perhaps CustomizeDependencies might resolve that though it works so am not touching the generator anymore!)

Now my next step is to wire up the test runner.

[gasparnagy]

Thx for the findings. Once you have the things working, would you please submit a PR with the documentation changes related to these? This would help others going through the same route...

[WBMPH]

I've made a little bit of progress but now stuck on the dependency injection. However I'm able to get all scenarios, all steps, start a scenario and get the result out of scenario. This all is integrated in our Blazor application:

So far this is what I did:

1. I created a custom generator plugin that extends the Nunit generator and adds a custom Attribute to the test class initialize method. This helps me scan the scenarios assembly more easily (more on that later)

It looks like this:

public class CodeGeneratorPlugin : IGeneratorPlugin
{
    public void Initialize(GeneratorPluginEvents generatorPluginEvents, GeneratorPluginParameters generatorPluginParameters,
        UnitTestProviderConfiguration unitTestProviderConfiguration)
    {
        generatorPluginEvents.RegisterDependencies += RegisterDependencies;
    }

    private void RegisterDependencies(object sender, RegisterDependenciesEventArgs registerDependenciesEventArgs)
    {
        registerDependenciesEventArgs.ObjectContainer.RegisterTypeAs<UnitTestGeneratorProvider, IUnitTestGeneratorProvider>("nunit");
    }
}

public class UnitTestGeneratorProvider : NUnit3TestGeneratorProvider
{
    protected internal const string INFO_ATTRIBUTE = "Simulation.Shared.ScenarioInformationAttribute";
    public UnitTestGeneratorProvider(CodeDomHelper codeDomHelper) : base(codeDomHelper)
    {
    }

    public override void SetTestClassInitializeMethod(TestClassGenerationContext generationContext)
    {
        base.SetTestClassInitializeMethod(generationContext);
        CodeDomHelper.AddAttribute(generationContext.TestClassInitializeMethod, INFO_ATTRIBUTE, generationContext.Document.DocumentLocation?.FeatureFolderPath, Path.GetFileName(generationContext.Document.DocumentLocation.SourceFilePath), generationContext.Feature.Name, generationContext.Feature.Description);
    }
}

2. I load in the generated scenarios assembly dll into my main application that will run the simulation and provides some additional services:

private static void LoadBDDTests()
{
    AppDomain.CurrentDomain.AssemblyResolve += (sender, args) =>
    {
        var assemblyName = new AssemblyName(args.Name);
        var path = Path.Combine(AppContext.BaseDirectory, "BDD", assemblyName.Name + ".dll");
        return File.Exists(path) ? Assembly.LoadFrom(path) : null;
    };
    BddAssemblies.Add(Assembly.LoadFile(Path.Join(AppContext.BaseDirectory, "BDD/Simulation.Scenarios.dll")));
}

3. I scan the assembly for scenarios. (I only later realized that the original feature file is included in the resources so step 1 might be obsolete.

ScenarioAssembly = Program.BddAssemblies.First();
Scenarios = ScenarioAssembly.DefinedTypes.Where(t => t.IsClass && t.GetCustomAttributes(true).Any(a => a.GetType() == typeof(TestFixtureAttribute)))
    .Select(info =>
    {
        var method = info.GetMethod("FeatureSetupAsync");
        var attribute = method.GetCustomAttribute<ScenarioInformationAttribute>();

        var scenarioData = new ScenarioData
        {
            TypeInfo = info,
            ClassName = attribute.Title,
            Description = attribute.Description,
            Location = attribute.FolderPath,
            Content = string.Empty
        };
        using var stream = ScenarioAssembly.GetManifestResourceStream($"{attribute.FolderPath.Replace("/", "\\")}\\{attribute.FeatureFile}");
        using var reader = new StreamReader(stream);
        scenarioData.Content = reader.ReadToEnd();
        var parser = new ReqnrollGherkinParser(new CultureInfo("en-US"));
        using var stringReader = new StringReader(scenarioData.Content);
        scenarioData.Document = parser.Parse(stringReader, null);
        return scenarioData;
    })
    .ToList();

4. As I want to list all scenarios + available steps I re-use the BindingRegistryBuilder like so:

ContainerBuilder = new ContainerBuilder(new BindingDiscoveryDependencyProvider(_serviceProvider)) { SkipLoadingProvider = true };
ObjectContainer = ContainerBuilder.CreateGlobalContainer(ScenarioAssembly, new JsonStringRuntimeConfigurationProvider("{}"));
BindingRegistryBuilder = ObjectContainer.Resolve<IRuntimeBindingRegistryBuilder>();
BindingRegistryBuilder.BuildBindingsFromAssembly(ScenarioAssembly);
BindingRegistryBuilder.BuildingCompleted();
BindingRegistry = ObjectContainer.Resolve<IBindingRegistry>();

5. To start a scenario:

ScenarioData data = Scenarios.First(); // Any scenario from Scenarios in step 3
var fixture = Activator.CreateInstance(data.TypeInfo.UnderlyingSystemType); // This is our testclass fixture
var testRunnerField = testRunnerField = _fixture.GetType().GetField(GeneratorConstants.TESTRUNNER_FIELD, BindingFlags.NonPublic | BindingFlags.Instance); // I want to hook into the events, so sneaky reflection here. Open to suggestions / alternatives but simplest way I could achieve.
var setupMethod = setupMethod = Data.TypeInfo.GetMethod(GeneratorConstants.TESTCLASS_INITIALIZE_NAME);
var setupTask = (Task)_setupMethod.Invoke(_fixture, []);
setupTask.Wait(); // Initialized the private testrunner field.

var testRunner = (ITestRunner)_testRunnerField?.GetValue(_fixture);

var executionEvents = TestRunner.FeatureContext.FeatureContainer.Resolve<RuntimePluginTestExecutionLifecycleEvents>(); 
executionEvents .BeforeTestRun += ... // some custom event handling.


var testMethod = (Task)data.TypeInfo.GetMethods().FirstOrDefault(m => m.GetCustomAttribute<TestAttribute>() != null)?.Invoke(_fixture, [])!;
testMethod.Wait();

So far so good. I can load the assembly, start it and get the result out it too (TestRunner.ScenarioContext.ScenarioExecutionStatus). Using the BeforeStep / AfterStep events I can track the progress of the test.

Right now I'm having some trouble trying to resolve services provided by my main application.

So far I've tried the following thought neither of them work. Open to suggestions / pointers...

A: I've tried adding the directly to the feature / testthreadcontext (scenario seems to be null?)

        TestRunner.FeatureContext.FeatureContainer.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IServiceA>());
        TestRunner.FeatureContext.FeatureContainer.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IDependencyB>());
        TestRunner.TestThreadContext.TestThreadContainer.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IServiceA>());
        TestRunner.TestThreadContext.TestThreadContainer.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IDependencyB>());

B: I tried registering them during the binding discovery:

    private class BindingDiscoveryDependencyProvider : DefaultDependencyProvider
    {
        private readonly IServiceProvider _serviceProvider;

        public BindingDiscoveryDependencyProvider(IServiceProvider serviceProvider)
        {
            _serviceProvider = serviceProvider;
        }

        public override void RegisterGlobalContainerDefaults(ObjectContainer container)
        {
            base.RegisterGlobalContainerDefaults(container);
            container.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IServiceA>());
            container.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IDependencyB>());
        }

        public override void RegisterTestThreadContainerDefaults(ObjectContainer testThreadContainer)
        {
            base.RegisterTestThreadContainerDefaults(testThreadContainer);
            testThreadContainer.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IServiceA>());
            testThreadContainer.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IDependencyB>());
        }
    }

C: I tried adding them to the global container created in 4

        ObjectContainer.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IScheduler>());
        ObjectContainer.RegisterFactoryAs(s => _serviceProvider.GetRequiredService<IBatchCache>());

[WBMPH]

I've been confused where the Unable to resolve service for x while attempting to error message was originating as I was under the assumption that BoDi would inject all required services however that error message does come from IServiceProvider.

I've now gone via the ugly static singleton shared IServiceProvider route instead to wire up my services but would like to investigate this further. I couldnt quite exactly find where the binding classes are instantiated.

[gasparnagy]

Yes, the Reqnroll DI error message would start with something like Interface cannot be resolved.

The binding classes are created on-demand when Reqnroll tries to execute the first binding method on them. The creation goes through the ScenarioContext.GetBindingInstance method that delegates the actual resolution to the ITestObjectResolver interface.

The BindingInvoker builds up a delegate that contains the ScenarioContext.GetBindingInstance call and whenever the delegate will be finally invoked.

It's hard to see from the code why you get the service provider error. If you have a chance to run it locally or at least get a full stack trace that would probably help.