[Bug]: Playwright not sharding tests evenly

[rohitkrishna094]

Version

^1.48.0(latest)

Steps to reproduce

Example steps to reproduce.

1. Copy this code into a file called tests/shard-test.spec.js

const { test, expect } = require('@playwright/test');

const arr = Array.from({ length: 2000 }, (_, i) => i + 1);

test.beforeAll(async () => {
    console.log('Before tests');
});

arr.forEach((num) => {
    test(`@smoke should check Math.pow functionality for ${num}`, async ({ page }) => {
        expect(num ** 2).toBe(Math.pow(num, 2));
    });
})

2. Also have fullyParallel set to true in your playwright config file
3. Then run the test using this command npx playwright test tests/shard-test.spec.js --shard=2/8
4. Look at the logs and you will notice that no tests run.
5. However on shard 1 it gets 500 tests allocated to it(when in fact each shard should get 2000/8 = 250 tests)

Expected behavior

• Each shard should get same amount of tests allocated to it.
• So in this example each shard should get 2000/8 = 250 tests each.
• And finally some shards should not exit immediately doing no work

Actual behavior

• Bug in playwright where tests are not sharded evenly
• Noticed that this happens if I added test.beforeAll in this file. If I comment that out, then it works fine.

Additional context

No response

Environment

System:

• OS: Windows 10 10.0.19045
• CPU: (8) x64 Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
• Memory: 6.76 GB / 15.88 GB

Binaries:

• Node: 20.16.0 - C:\Program Files\nodejs\node.EXE
• npm: 10.8.2 - C:\Program Files\nodejs\npm.CMD

IDEs:

• VSCode: 1.94.0 - C:\Users\Owner\AppData\Local\Programs\Microsoft VS Code\bin\code.CMD

Languages:

• Bash: 5.1.16 - C:\WINDOWS\system32\bash.EXE

npmPackages:

• @playwright/test: ^1.48.0 => 1.48.0

[rohitkrishna094]

This seems like an issue from few versions ago since the new algorithm is to shard files as opposed to tests
#21497
Comment

But even that seems to not be the correct statement since if it was true, then in my example the 1st shard should get all the tests while the remaining shards should get 0 tests. However seems like there is some form of hybrid shading algorithm at play here.

Besides that, there has to be a way to shard tests evenly even if I am using a single file because there are many usecases where users might generate tests dynamically and still have beforeEach or beforeAll in their test files.