Feature Hooks

[Gapmeister66]

Scenarios are grouped into features and typically share some common configuration state. A common problem is to set up and tear down common configuration state before and after a feature is executed; currently it is not possible to do this. A workaround would be to reset such state in a 'background' or 'before' step, but these handlers are executed before each scenario, not each feature. If the handler is itself expensive or slow, these workarounds will slow down total execution time.

I'd like to request @BeforeFeature and @AfterFeature tags. These tags execute at the expected time consistent with their names.

I've tried using background steps, JVM hooks & before and after tags. All of these mechanisms have drawbacks and do not provide a clean solution addressing time constraints problems. If configuration switching was always fast there would not really be a problem. There are plenty of examples of complaints and workarounds on the internet but the problem never seems to be addressed and seems to be ignored or sidelined.

Typical application would be to set (@BeforeFeature) or reset (@AfterFeature) a configuration value and restart a service, container or pod that consumes a non trivial period of time.

[aslakhellesoy]

Every scenario gets a new instance of stepdef classes, which is the "scope" of the step definitions and scenario hooks.

What "scope" would feature hooks run in?

[Gapmeister66]

That's interesting... I rarely put state into the step definitions (except service stub singletons). If I absolutely need to, I inject a singleton "state" object, but then I generally clear the state in the object before each scenario using @before. So for me, the scope is not of a concern, I just need something to run before and after each feature and I would not intend to access any shared state in the proposed hooks.

Just to add, I use cucumber-guice, which is how my shared objects are injected. I can now see why the scope you mention is important if IOC is not used.

May I suggest something (apologies if this already exists)? Could you provide some kind of switch that provides new instances in addition to scenario scope, e.g. Singleton/Feature/Scenario, defaulting to 'Scenario' (current behaviour). For those of us not taking advantage of the default 'Scenario' scope, we could explicitly change to 'Singleton' scope.

However, this would not help me unless the proposed hooks are implemented, but having such a switch makes things explicit and would be more efficient.

[aslakhellesoy]

We still need a context, even if you are not using it.

We can require those hooks to be static methods (i.e. no context).

WDYT @mpkorstanje ?

[Gapmeister66]

Things are clearer to me now. Cucumber is essentially tearing down state after each scenario. It would be really useful if we could allow developers to choose when this happens and how it happens, but also to leave the defaults in place for those new to cucumber and for those using the current scoping rules.

[mpkorstanje]

Fundamentally this the same problem as #515. And a lot of the problems that were blocking #515 have been solved in v5 so we could schedule #515 for v6.

The added complication is that feature hooks would need to be executed around features. Because scenarios and thus features are executed in parallel the context in which feature hooks are executed can not be static. We could introduce a separate feature context that exists for the span of all scenarios in the feature but since this context would be inaccessible from within the scenario context I don't see a point in doing so.

Looking at the root cause of the request -the need to reuse some limited and expensive resource between consumers- I can't help but think of a Connection Pool or more generally speaking an Object Pool. I would suggest that @Gapmeister66 implements something similar for his system under test.

I also believe people have had a similair problem with parallel execution and remote web drivers. So this might be a blog worthy topic.

The general setup might be something like:

1. A collection of features. Each tagged at feature level with the needs of the system.

@system-configuration-A
Feature: First feature
  
  Scenario: 1
    ...
  Scenario: 2
    ...

@system-configuration-B
Feature: Second feature
  
  Scenario: 3
    ...
  Scenario: 4
    ...

2. A before hook that requests a system under test from the pool using the annotations.

@Before
public void setup_system_under_test(Scenario scenario){
      system = System.start(scenario.getSourceTags());
}
@After
public void setup_system_under_test(Scenario scenario){
      system.stop()
}

Note that:

1. Tags are inherited. A scenario has all tags of its parent feature.
2. .stop() does not shutdown the SUT, it only marks it as reusable.
3. With a with a pool size of 1 the SUT will be reused between scenario 1&2 and 3&4. Because a new configuration is requested in the third scenario the SUT will be shutdown and restarted.

I don't think using tags is the right way to do this. I would setup the the SUT using a step, but using hooks with tags was easier to explain.

[mpkorstanje]

I'll close this for now.

Cucumber by design runs each pickle (i.e. compiled scenarios) in its own context. This ensures that each test is executed independently from all others. This is a design principle that can be found in most test frameworks.

While we could certainly make this behaviour optional this comes at a cost:

1. It would first of all be most surprising to encounter a framework that did not isolate its tests. This is not a good property of a test framework and would make Cucumber objectively worse.

2. It would come at a significant implementation and maintenance cost. While this effort would make it possible to solve a problem, it would so poorly. The problem would be much better solved by implementing the object pool design pattern on the user side.

So I don't think this feature is worth investigating further.

[Gapmeister66]

Thank you for the feedback.

I work with both Java and Groovy Cucumber implementations. Solution (2) is not available in Groovy to my knowledge because it handles tags differently.

On a separate note, whilst I generally agree principles are useful and help with the general case, there are always exceptions to the rule. The principle: 'a test is executed independently of others' assumes that setup/tear-down is an instantaneous or at least a very quick action. In our case, resetting a k8s configuration value before each scenario would take 30 seconds due to pod restarts, and then to reset after the scenario would take another 30 seconds, so in practical terms it simply wouldn't be workable. Ironically, it violates the test principle: 'tests should run quickly'. We have a very large test bed.

New step definition object per scenario assumes that all state is held in step definitions, whereas in reality it is likely to be held in a DB. I appreciate that @before works around this.

We will have to re-evaluate our current solution and determine whether Cucumber can support our needs or whether we can just live with the pain.

[mpkorstanje]

I work with both Java and Groovy Cucumber implementations. Solution (2) is not available in Groovy to my knowledge because it handles tags differently.

You're saying this doesn't work?

Before() { it ->
    def names = it.getSourceTagNames()
}

That would be surprising to me. You should probably file a bug report with cucumber-jvm-groovy then.

On a separate note, whilst I generally agree principles are useful and help with the general case, there are always exceptions to the rule.

This is true. However from the perspective of a framework it is impossible to enumerate all exceptional situations and certainly be prohibitive to attend to all. Remember this is an open source project - some assembly required.

The principle: 'a test is executed independently of others' assumes that setup/tear-down is an instantaneous or at least a very quick action. In our case, resetting a k8s configuration value before each scenario would take 30 seconds due to pod restarts, and then to reset after the scenario would take another 30 seconds, so in practical terms it simply wouldn't be workable.

This is something that you can control though. Which aspects of implementing the object pool pattern are problematic for you?

We will have to re-evaluate our current solution and determine whether Cucumber can support our needs or whether we can just live with the pain.

If you absolutely must solve your problem using framework provided hooks TestNG might be an option to consider. It has the ability to organize test classes into groups and execute hooks before and after groups.