can autoverify be set only for the first verify?

[ci-vamp]

i see autoverify can be set on a per test or global basis. there is also the feature of disabling in CI environments which is nice.

what i would like is to auto verify just on the first time (when there is no verified file). is my understanding correct that enabling autoverify globally will accept snapshots even if an existing verified file exists for that test? if so, is there a way to make it only for first verify?

[SimonCropp]

no this is not currently supported.

the theory is that people should eyeball the initial version

[ci-vamp]

our use case is running tests against several data sets (controlled via env var). it has become tedious to gather the snapshots for all 3 data sets by going into the unit test settings of resharper, changing the env var, running the tests for the next data set then verifying and committing the new snapshots.

as an alternative i wrote a script that iterates over the data set labels and runs the tests with auto-verify. this way we still validate the new snapshots but all at once for a new test and through the git diff.

however, once a snapshot has been reviewed and committed i would like to fail if it differs and fallback to the behavior as if autoVerify is disabled.

we can establish a process around this (if you commit with a snapshot that has been modified treat it as a potential red flag). but as an extra safeguard it would be nice to prevent this situation happening at all.

do you think its possible to do this through the OnFirstVerify hook? in pseudocode:

OnFirstVerify((snapshot) => {
  snapshot.AutoVerify();
});

[SimonCropp]

going into the unit test settings of resharper

I would have done it with module init

[ModuleInitializer]
public static void Init()
{
  //VerifierSettings.AutoVerify();
}

then uncomment when i want to AutoVerify

but i dont really understand the friction here.

My workflow:

1. Add a new test
2. Run
3. the initial failure causes a diff to show
4. I hit the "Accept all open" Hotkey. See screenshot below
5. the new verified file is created

What is your workflow?

[ci-vamp]

ya sorry for the miscommunication.

we have 3 (and potentially more in the future) data sets that we run the tests against. each represents a certain client context, altogether giving us confidence in the system.

we switch these data sets via an environment variable which a global setup uses to load the respective data set.

we have a global config that will save the snapshots under Snapshots/<Dataset Label>/ dir since each data set has similar but distinct (in terms of values) snapshot results

the non autoverify workflow looks like:

1. write tests
2. go into Build, Execution, Deployment > Unit Testing > Test Runner > modify the environment variable value
3. run tests
4. review and accept the snapshots
5. repeat step 2-4 for each of the remaining data sets

this gets tedious with the 3, and moreso if we increase the data sets, because the only way (that i can tell) to switch the env var is to go into those nested settings to set the env var manually each time.

the autoverify workflow looks like:

1. set auto verify globally in the module init
2. write tests
3. run script that iterates over the env var values and runs dotnet test for each data set value
4. review the auto-verified snapshots as git diffs
5. commit becomes the "accept" step

this second approach is a lot smoother but because there is no way to only auto verify for first snapshots it presents a risk because even modified snapshots will be considered accepted. we can control this by monitoring for "modified" diffs vs "created" but it would be preferable to control this at the verify level so we can remove the human potential for error.