[Bug?]: Scenarios not cleaned up after a test fails -> can cause cascading test failures

[jason-curtis]

What's not working?

According to https://docs.redwoodjs.com/docs/testing/#which-scenarios-are-seeded :

During the run of any single test, there is only ever one scenario's worth of data present in the database

Sometimes that's not the case - I'm finding that when a test case fails, the related scenario is left over in the DB. This causes subsequent tests to fail, either due to uniqueness constraints being violated, or when they assert how many rows there should be and find too many rows.

How do we reproduce the bug?

1. create an app with a prisma model (say, user), using a postgres DB
2. Edit your test file users.test.ts so that there is a test that will throw an error and other tests after it. For example:

import { users } from './users'
import type { StandardScenario } from './users.scenarios'

describe('users', () => {
  scenario('busted test', async () => {
    throw new Error()
  })
  scenario('returns all users - fails due to leftover setup from previous test', async (scenario: StandardScenario) => {
    const result = await users()

    expect(result.length).toEqual(Object.keys(scenario.user).length)
  })
})

3. yarn rw test - should result in only one test failure, but both tests will now fail due to leftover DB entities from the busted test interfering with the second test. Depending on your user model, I've seen two possible errors:

• if your user model has uniqueness constraint (on the email column, say), then that constraint will cause scenario setup to fail:

  ● users › returns all users
  
    PrismaClientKnownRequestError: 
    Invalid `getProjectDb()[model].create()` invocation in
    /home/jason/code/farm-data/node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js:202:64
  
      199     createArgs(scenarios)
      200   )
      201 } else {
    → 202   scenarios[model][name] = await getProjectDb()[model].create(
    Unique constraint failed on the fields: (`email`)
  
      at ai.handleRequestError (node_modules/@prisma/client/runtime/library.js:126:6775)
      at ai.handleAndLogRequestError (node_modules/@prisma/client/runtime/library.js:126:6109)
      at ai.request (node_modules/@prisma/client/runtime/library.js:126:5817)
      at l (node_modules/@prisma/client/runtime/library.js:131:9709)
      at seedScenario (node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js:202:36)
      at Object.<anonymous> (node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js:107:28)
  

• if there are no uniqueness constraints other than the primary key, you will just get an error because there are too many user entities lying around in the database:

  ● fieldsAsSuperuser › returns all users

    expect(received).toEqual(expected) // deep equality

    Expected: 2
    Received: 4

      38 |     const result = await users()
      39 |
    > 40 |     expect(result.length).toEqual(Object.keys(scenario.user).length)
         |                           ^
      41 |   })
      42 |
      43 |   scenario('creates a user', async (scenario: StandardScenario) => {

      at toEqual (api/src/services/users/users.test.ts:40:27)
      at Object.<anonymous> (node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js:108:22)

What's your environment? (If it applies)

System:
    OS: Linux 5.15 Ubuntu 20.04 LTS (Focal Fossa)
    Shell: 3.6.0 - /usr/bin/fish
  Binaries:
    Node: 20.11.1 - /tmp/xfs-3d851e06/node
    Yarn: 4.1.0 - /tmp/xfs-3d851e06/yarn
  npmPackages:
    @redwoodjs/auth-supabase-setup: 7.0.6 => 7.0.6 
    @redwoodjs/cli-storybook: 7.0.6 => 7.0.6 
    @redwoodjs/core: 7.0.6 => 7.0.6 
    @redwoodjs/project-config: 7.0.6 => 7.0.

Are you interested in working on this?

• I'm interested in working on this

[ahaywood]

Hey @jason-curtis thanks for taking the time to file an issue. I'll get someone on the team to take a look!

[Josh-Walker-GM]

Hey @jason-curtis! Thanks for the details here.

I just tried to reproduce this with our test project fixture. I used your example scenario code and got a random postgres database from railway (so it would be slower than my localhost connection). I wasn't able to see the results where the second one fails because of the first deliberate failure.

Could you share the scenario code you had too maybe that'll help me?

I also didn't see the behaviour with sqlite.

[Josh-Walker-GM]

Ah my bad! I realised I wasn't testing with the same version you specified in your report. If I switch to v7.0.6 then I see the failure you describe here. I tested on v8.0.0 and it passed.

Let me see if I can hunt down the version that fixed it.

[Josh-Walker-GM]

Okay looks like this might be a duplicate of #10068 which was then solved in #10112 which went out in v7.0.7. Looks like you're just 1 patch version away from the fix haha!

Could you upgrade to v7.0.7 and confirm? If that fixes things for you then we can close this issue.

[jason-curtis]

Upgrading fixed the issue! Thank you for looking into this @Josh-Walker-GM !