Add "discovery" as a kind for TestRunProfiles

[eleanorjboyd]

The testing API currently allows for TestRunProfiles with 3 different options for kind: run, debug, or coverage. With the Python extensions upcoming changes to custom args which expands functionality, we would need a discovery kind of profile (see proposal for custom args here). Looking to have this profile be configurable the same way the run button is in the test view when you have multiple configs. Have the discover button (the refresh button) have a dropdown that lists the different configuration options, a default, and a click through option to edit the configs.

[connor4312]

I'm not clear on why we need a "discovery" test run profile kind. What kind of results would be reported from a discovery test run? Why does the refreshHandler not already solve this?

[eleanorjboyd]

The "discovery" run kind would allow the user to have different args configured for their test discovery. It wouldn't necessarily be a "run" event as much as it would just trigger the same "discover tests" event which is triggered by the "refresh" button in the test explorer. From my understanding the refreshHandler does not pass in config I could somehow reference to understand which discovery config in the settings.json the user is referring to.

[connor4312]

This seems like we're trying to squeeze test profiles into a place where they don't really belong.

Would it be acceptable for the extension to instead show a quick pick in its refreshHandler if there's >1 profile configured so that the user can pick which profile(s) to refresh that way?

[eleanorjboyd]

Which is the refreshHandler? That seems like it would be fine if the refreshHandler quick pick / selection was in an intuitive place for users.

[connor4312]

The refreshHander is defined on the TestController and is what's invoked when the "refresh" button in the Test Explorer view is clicked.

[eleanorjboyd]

Yes this sounds good! Good point that it is not a TesetRunProfile but instead a different style of profile since discovery is a different process.

Would it be possible to have UI where a user can select a default for their discovery profile? My worry is the flow is not as clean if users must click "refresh" -> "select profile" to run every time instead of just "refresh" and then change the default if they do want to switch up their discovery command. We could also opt to do the quickpick for a MVP and make changes later based on what users actual usage looks like.

[connor4312]

From discussion: we can have the refreshHandler refresh all configurations, and an extension-contributed command can run discovery with individual configurations. If telemetry usage shows people are using the per-configuration discovery command often, we can consider making some API changes for this.