-
Notifications
You must be signed in to change notification settings - Fork 338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Programmatic skip scenario feature #2502
Programmatic skip scenario feature #2502
Conversation
Signed-off-by: Piotr Nestorow <piotr.nestorow@systemverification.com>
Any comments regarding the above functionality proposal? |
I need to think this through @PiotrNestor . The first thing that hits me is the impact on statistics especially for table driven execution. And a couple of other scenarios. Will list them in some time. More importantly, how would this feature help? Genuinely interested in the use case |
There are a number of use cases for the feature of programmatically skipping a scenario:
|
hi @PiotrNestor , apologies for the delay. As a feature I can understand this can be useful. I am interested to know your thoughts on the below:
The benefits: it keeps the logic of planning your test execution outside the test suite and your test suite need not connect to the test planner (useful if you are running it in CI). the cons: It makes the orchestration a bit tricky.
The reason to bring this up is that this could be a case to consider if we are programmatically skipping. Beyond these, I think we also need to think about bringing in visibility into reasons for skipping the scenario in the reports, but that is a separate thing altogether. In summary - I guess I am ok with this feature, and the first cut of this would be to ignore a scenario altogether without worrying about the implications on table driven v/s normal execution. |
@sriv |
This feature is not good enough for the integration with a test management tool selecting a list of scenarios to run. Problems:
The possible implementation needs then:
If skip scenario during execution is available the above integration is simple. |
Have you seen https://manpage.gauge.org/gauge_list before? Basically I suppose if this list is good enough then comparing them with a run list to come up with a final list should be possible with a few script. I am reminded of an issue where For example I was exploring the option to do |
I just realized that So, if you still prefer the solution of skipping scenarios programatically, I guess we can go with it. In an ideal world, I would love to add support for a plugin that determines the run list for a suite, but that is a larger change. |
Signed-off-by: Piotr Nestorow <piotr.nestorow@systemverification.com>
@sriv |
Signed-off-by: Piotr Nestorow <piotr.nestorow@systemverification.com>
Signed-off-by: Piotr Nestorow <piotr.nestorow@systemverification.com>
@sriv When the above are accepted I can proceed with the relevant gauge-dotnet and gauge PR. With the programmatic skip: on the scenario implementation side it's possible to:
|
Signed-off-by: Piotr Nestorow <piotr.nestorow@systemverification.com>
I think your dependent commits have been merged now, so you can probably rebase this off master and use latest protos from go. Might be easier to start again with go.mod/sum rather than try and rebase commits. |
Signed-off-by: Piotr Nestorow <piotr.nestorow@systemverification.com>
Signed-off-by: Piotr Nestorow <piotr.nestorow@systemverification.com>
@sriv The main points of the skip scenario sequence:
|
Benchmark Results
Notes
See Workflow log for more details. |
@sriv |
@sriv ping! |
Hi! @sriv Do you have any opinions regarding this PR? We'd like to know if it can be accepted, or if it needs to be improved/redesigned or if it should be considered as rejected altogether? By accident I found these two requests as well, which I think will be solved by this PR (we only updated Guage & the DotNet-runner, to get it working in Java then the Java-runner also needs the update, obviously). getgauge/gauge-java#691 Best regards, |
sorry about the delay @PiotrNestor @jensakejohansson - been travelling a lot most summer and work is super tight. I have approved this, but I guess this PR needs a rebase. |
Apparently this was unreleased, so it's in |
@sriv
This is a discussion and proposal for a programmatic skip scenario feature.
Before providing the actual code changes I'd like to discuss, review and agree on how this should be implemented in the different gauge components.
The proposed programmatic scenario skip feature is similar to the C# NUnit programmatic test ignore feature.
Scenario skip should be possible during execution of a correct scenario in a hook or step method.
Note that this is different from the present skip of a scenario during the validation phase when some step is missing or invalid.
Although the idea is that the outcome would be similar where the scenario is reported as skipped.
Implementation proposal:
Our question is now if the above is clear enough to proceed with actual updates implementation?