Tests under subfolders of features folder are not executing for version greater than 4.0.0

[magarwal19]

Hi Team,
On the latest version when we run command "npx bddgen && npx playwright test"
then the .features-gen folder created do not create features folder and the command gets fail stating "Error: No tests found"
while if we downgrade to version 4.0.0, then features folder is created properly and able to execute tests

[vitalets]

Hi @magarwal19 ! could you provide more details on your directory structure?

[magarwal19]

@vitalets
it is like below:

[vitalets]

I've got it. This caused by #40 that simplifies output directory structure utilizing nearest common parent of feature files.
So now you will get:

.features-gen
  flow1
    feature1.feature.spec.js
  flow2
    feature2.feature.spec.js

instead of:

.features-gen
  features
    flow1
      feature1.feature.spec.js
    flow2
      feature2.feature.spec.js

Is it something that does not suit your project? I mean could you set outputDir to .features-gen in config - and it should work?

[magarwal19]

I was not setting up output directory explicitly in playwright config file as i think it by defaults takes .features-gen

Problem with new version is that the run command
npx bddgen && npx playwright test

is failing stating no tests are found to run

[magarwal19]

@vitalets i tried adding outputDir, but still getting same error
However, if i change the outputDir to outputDir : ".features-gen/features"
then i am able to run tests

[vitalets]

I was not setting up output directory explicitly in playwright config file as i think it by defaults takes .features-gen

You are right. I just wanted to double check your settings.

So I've tried to reproduce it in playwright-bdd-exmple dir-structure-issue-44 branch.

Having the same directory structure:

And config:

const testDir = defineBddConfig({
  paths: ['features/**/*.{feature,feature.md}'],
  require: ['steps/*.ts'],
  importTestFrom: 'steps/fixtures.ts',
});

When run npx bddgen && npx playwright test all tests pass.

Could you clone playwright-bdd-exmple on that branch and run npm test?

[magarwal19]

Hi @vitalets
i checked out the repository and able to run tests with
"npx bddgen && npx playwright test" command

It seems i found the issue..
I was running a specific feature file in my local and there it fails saying no file is found.

I am able to replicate it with the repository you have shared..

Can you please try running below command:
npx bddgen && npx playwright test features/flow1/homepage.feature

[vitalets]

Can you please try running below command:
npx bddgen && npx playwright test features/flow1/homepage.feature

Passing .feature files directly to Playwright CLI will not work, as Playwright expects js/ts files. But Playwright is smart enough to search for **/*.spec.js in provided argument. So you can just replace features/... with .features-gen/... to run a particular file:

npx bddgen && npx playwright test .features-gen/flow1/homepage.feature

or more accurately:

npx bddgen && npx playwright test .features-gen/flow1/homepage.feature.spec.js

[magarwal19]

Hi @vitalets

passing .feature file use to work earlier on old version.

with new changes if i change my run command to

npx bddgen && npx playwright test flow1/homepage.feature

then this works...

But when we were using old way of giving complete paths "features/flow1...."
then it was easy as console helps to generate path by using tab keys..

also .features is ideally created after running our run command so ideally user will not be aware about its path

[vitalets]

I've got the point. Looking deeply on how Playwright handles passed arguments, I found that it converts them into RegExp and applies to absolute file paths.
For example, when calling Playwright with .feature file:

npx bddgen && npx playwright test features/flow1/homepage.feature

It builds regexp like this:

/features\/flow1\/homepage.feature/gi

And applies it to all files from testDir (that is .features-gen):

.features-gen/features/flow1/homepage.feature.spec.js
...

When directory structure was the same as in features, regexp check was passing as generated files contain whole original filename. After #40 generated directory structure was simplified to the nearest parent dir and regexp check stopped passing.

Generally I like that "trick" with passing .feature files directly to Playwright. I think of introducing config option outputDirStructure: 'preserve' | 'simplify' to fit all needs. Preserving directory structure is also useful for screenshots storing, as they saved next to generated test files.

@magarwal19 thanks for bringing all this ideas :)

[magarwal19]

That would really be helpful!
thank you for prompt response!

[vitalets]

@magarwal19 while working on this fix I realized a workaround how we can pass .feature files to Playwright cli with current codebase. We could set outputDir to something ending with features, for example outputDir: '.features':

const testDir = defineBddConfig({
  outputDir: '.features', // <- not .features-gen
  paths: ['features/**/*.{feature,feature.md}'],
  require: ['steps/*.ts'],
  importTestFrom: 'steps/fixtures.ts',
});

Then run command with .feature file:

npx bddgen && npx playwright test features/flow1/homepage.feature

Substring features/flow1/homepage.feature will match test file path .features/flow1/homepage.feature.spec.js and tests will run.

[vitalets]

Concerning outputDirStructure I'm still thinking of approach. I should admit that output directory simplification introduced in #40 was not the best choice. Playwright heavily relies on directory structure for storing screenshots/snapshots, and we should try to keep it as stable as possible.

For example, the following features structure:

features
  flow1
    feature1.feature

produces the following output dir:

.features-gen
  feature1.feature.spec.js
  feature1.feature.spec.js-snapshots
    snapshot.png

Just adding a new feature file can cause full re-structure of outputDir that will invalidate all saved screenshots. It will be a headache for developers to find out why all screenshot tests are suddenly failed:

features
  flow1
    feature1.feature
  flow2
    feature2.feature

output dir:

.features-gen
  flow1
     feature1.feature.spec.js
  flow2
     feature2.feature.spec.js
  feature1.feature.spec.js-snapshots
    snapshot.png  // <-- screenshot is not valid as it's located not next to test file

[magarwal19]

yes agree...
with current directory changes, when we open html report, then it does not show the screenshots in report as playwright not able to identify due to directory changes.

[vitalets]

Fixed in 5.2.0. Introduced new config option featuresRoot that solves this and other issues related to outputDir structure.

Unfortunately this is possibly breaking change for screenshot testing depending on project structure.
Added one more disclaimer in changelog.

@magarwal19 could you check on your side with 5.2.0 that everything works correctly?

[magarwal19]

Hi @vitalets
tested it on new version 5.2.0
it is working as expected now and I am able to run specific test. Also screenshots and videos are properly mapped in default html and allure reports

[vitalets]

Thank you!