ReqNRoll standalone runner and dynamic feature definitions

[jonmotos]

I'm not sure if this is a new idea per se, but am interested in your knowledge of our proposed use case.
We are hoping to use ReqNRoll in such a way where we can:

1. Publish the test assembly and it's dependencies to a standalone DLL, ideally run from CLI
   I believe this is doable with dotnet test out of the box, using the mstest runner? I am unsure if this plays well with ReqNRoll however.
2. Edit the feature definition files at runtime, to allow both for environment-specific configurations to be read-in, as well as adding and removing scenarios as desired given the specific environment.
   Perhaps this is better accomplished by reading in configuration files at runtime, and running subsets of tests using the current filtering mechanisms?

The specific case is that we have a complex set of applications with varying configurations, such as URL, hostname, port, etc. We want to be able to execute a command line task to verify specific behaviors of the system as a whole, typically subsets of defined features.

Could anyone share any thoughts or feedback on this idea? Perhaps there is some more suited tool, or ReqNRoll is entirely suited for this via some specific methods, I am unsure!

Thank you!

[tulavalle]

Não tenho certeza se esta é uma ideia nova em si, mas estou interessado em seu conhecimento do nosso caso de uso proposto.Esperamos usar o ReqNRoll de tal forma que possamos:

1. Publique o assembly de teste e suas dependências em uma DLL autônoma, idealmente executado a partir da CLIEu acredito que isso é factível com dotnet teste fora da caixa, usando o mstest runner? Não tenho certeza se isso joga bem com o ReqNRoll no entanto.
2. Edite os arquivos de definição de recursos em tempo de execução, para permitir que as configurações específicas do ambiente sejam lidas, bem como adicionar e remover cenários conforme desejado, considerando o ambiente específico.Talvez isso seja melhor realizado lendo em arquivos de configuração em tempo de execução e executando subconjuntos de testes usando os mecanismos de filtragem atuais?

O caso específico é que temos um conjunto complexo de aplicativos com configurações variadas, como URL, nome do host, porta, etc. Queremos ser capazes de executar uma tarefa de linha de comando para verificar comportamentos específicos do sistema como um todo, normalmente subconjuntos de recursos definidos.

Alguém poderia compartilhar algum pensamento ou feedback sobre essa ideia? Talvez haja alguma ferramenta mais adequada, ou ReqNRoll é totalmente adequado para isso através de alguns métodos específicos, não tenho certeza!

Obrigado!

Hi @JonAdamsFromNC, I've been using Specflow for a few years, in quite complex projects, for backend and frontend testing and I've already tested the same projects with Reqnroll, and the result was equally satisfactory. I can share my experience:

1. I have already tested Reqnroll with Runner from MSTest, NUnit and xUNit. They all worked well, the only difference really was the syntax of some points that I needed to adjust according to the documentation for each Runner.

2. I use the BDD approach in the development cycle, where the feature is just one of the process outputs, which allows for live documentation. In my opinion, it doesn't make sense for the feature to be edited at run time, that is, to address this part of environmental flexibility in the feature. Writing in domain and behavior language, the feature offers the necessary flexibility to execute the scenario automated or not, in different environments and devices. The specific test data settings (data mass) for each environment deal with the data mass. I often use the approach of cleaning the data and populating the entire database with each run. To populate I usually use sql scripts or populate via API, through seeds jem json files. This mass is treated according to the environment through a variable that I place in a json file with the environment settings, thus making it more flexible to run tests in any environment (host name, port, url, among others). To run via CLI command, I use tags on the feature and/or each scenario and, through the tags, we can filter which scenarios and/or features should be executed at a given time.

I don't know if I was able to help with these points, but it has worked very well, maintaining responsibility for the documentation feature, but allowing full flexibility in relation to environments and test sets being executed. And this works locally or in a pipeline, where the configuration json file has environment variables for each environment in which the test will be executed.

The only point that I haven't yet migrated my personal study projects to Reqnroll is that I need the HTML Report, in the case of Specflow, LivingDoc, but other than that the tool has just been launched but it is very powerful and I am confident that it will become even more shortly! So I really encourage its use out there! And you can be sure that having an HTML report I retire Specflow and migrate to Reqnroll.

[gasparnagy]

Thank you for the detailed answer, @tulavalle!

I do the same basically what @tulavalle mentioned if I need to run my tests with different configurations. I call this "set env - run test" strategy, maybe we should write a separate guide for it in the docs. But basically it works like this:

• The automation code loads the configuration (e.g. which browser to use) from an environment variable (of course with some defaults) and acts accordingly.
• During execution (e.g. on the build server), I call it in a way:
  • set env var to the desired configuration
  • run the tests with dotnet test (or other ways)

In PowerShell for example this is as simple as: $env:BROWSER_TARGET = 'Chrome'; dotnet test. You can even include/exclude some scenarios. E.g. if you tag all scenarios that are not supported by Chrome with @notsupported:Chrome, you would need to run:

$env:BROWSER_TARGET = 'Chrome'; dotnet test --filter "Category!=notsupported:Chrome"

Note: I move this discussion to the Q & A category

[jonmotos]

This is fantastic background, thank you so much for outlining your process @tulavalle!
In my case I think we can fully follow your process and effectively read in the runtime info as configuration data. I like this a lot.

And @gasparnagy, using the tags to pick and choose scenarios I think will be very effective.

So I think I only have one follow-up question as far as "runtime" data goes - it is true that defining or redefining scenarios is not strictly necessary at runtime, and I think this is very workable.

However, is it possible to add new scenario outline test-cases at runtime? In other words, let's say that we have a well-defined behavior expected for case a,b,c. However we discover that we need to add case "d", specifically. Or, for test-setup "1", we want a,b,c, and test-setup "2", we want d,e,f. More specifically this would be environment-specific data that cannot be shared, and should not be located with our source code.

MSTest behaves this way when using [DynamicData], and for example loading testcases from an excel spreadsheet say, or in our case we would be loading each test case from directories (one test case per directory).
I just saw the discussion for supporting DynamicData with MSTest, and I do think this would be very helpful. Although I'm not sure if it is related.

If this does not exist already, I would be happy actually to look at this runtime DynamicData behavior, @gasparnagy if you see this as something that is achievable!

Thank you all!

[gasparnagy]

@JonAdamsFromNC For that need, please check the ExternalData plugin. (You can replace the external data between executions, so with that you can achieve an almost dynamic behavior.)

[jonmotos]

Amazing. Thank you for your help @gasparnagy! I am excited to put the tool to use.