Focus automagically switches to Testing Panel

[sschneider-ihre-pvs]

Issue Type: Bug

When the extension boots up or detects change it switches to Testing Panel although autorun is off.

Extension version: 4.2.1
VS Code version: Code - Insiders 1.64.0-insider (7ae7f9d40b82a0f1a49d8ad180d27e16812ad95e, 2022-01-18T05:16:47.915Z)
OS version: Windows_NT x64 10.0.19042
Restricted Mode: No

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz (16 x 2904)
GPU Status	2d_canvas: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: enabled opengl: enabled_on rasterization: enabled skia_renderer: enabled_on video_decode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.75GB (4.76GB free)
Process Argv	--crash-reporter-id 5f9e326c-2a01-458b-95a4-f411641d6f56
Screen Reader	no
VM	0%

A/B Experiments

vsliv695:30137379
vsins829:30139715
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyl392:30422396
pythontb:30258533
pythonvspyt551cf:30291413
pythonptprofiler:30281269
vshan820:30294714
pythondataviewer:30285072
vscod805cf:30301675
pythonvspyt200:30323110
vsccppwt:30421239
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30404738
wslgetstarted:30421357
vscop804:30404766
vs360cf:30404996
vsclayoutctrc:30405799
azactmsalcf:30409288
vsccppwtstudy:30421978

[bruno-brant]

As far as I can tell, this behavior started on the insiders release on last Friday (Jan 14th). Everything that causes the test to run (any changes on any files, for instance) will cause vscode to switch focus to the testing panel.

My guess, since the extension wasn't updated, is that something changed with vscode that is causing this behavior. However, other test runners don't seem to have the same issue.

[connectdotz]

Indeed vscode behavior has changed in the latest insider version. This extension updates test items when source file changes or background run results are detected. As you guys correctly diagnosed there is no update from the extension, therefore, most likely the change is from vscode 🤔

cc @connor4312 any guess of what could cause this behavior change, and can it be turned off?

[connor4312]

Yes, this was added in microsoft/vscode#140596.

Perhaps a good heuristic--until we have proper autorun in the API--would be to only show the test view for test runs created via the UI, not the API. Thoughts?

[bruno-brant]

Perhaps a good heuristic--until we have proper autorun in the API--would be to only show the test view for test runs created via the UI, not the API. Thoughts?

IMHO makes a lot of sense, from a user standpoint, at least.

[connectdotz]

would be to only show the test view for test runs created via the UI, not the API. Thoughts?

Hmm... regarding tests created via UI:

1. if triggered by action from TestExplorer view, already there, no need to show test view again
2. if triggered by the editor gutter menu, the test status is already visible in the editor thus maybe not everybody would appreciate the auto switch to the test view?

I am sure there are use cases that auto-showing test view is desired, but it might be very extension/context/user specific and would be hard to generalize for a universal solution... Besides, there is already a command to show test view that extension/user can use today, I personally think no need to do anything extra regarding this feature. But if vscode really wants to, maybe make this show-test-view-on-test an opt-in setting, i.e. off by default, so it's at least backward compatible.

[connectdotz]

Just tried the latest insider (2022-01-20T06:33:46.414Z) - still behaves the same 😞 . I am worried the upcoming release will cause significantly degraded UI experience for some users. @connor4312 what do you suggest we do about it?

[connor4312]

if triggered by the editor gutter menu, the test status is already visible in the editor thus maybe not everybody would appreciate the auto switch to the test view?

There has been some back and forth on this. We've gotten feedback from users confused, especially if tests ran very quickly. Since the Debug view behaves the same way (showing by default when a session starts, with an option to hide it) I think having the testing view match it makes sense. When unintrusive, I tend to favor behavior that makes things more obvious to less experienced users, as more experienced users have the knowledge to change settings if they dislike certain behavior.

I've updated the behavior based on our discussion in microsoft/vscode@fb1eac3. It'll be in the next Insiders.

[connectdotz]

@connor4312 Thanks for the explanation, it's a lot clear what prompted this change and why you choose to mimic debug view behavior. However, I would like to point out a critical difference between debug and test views: in debug process, users would usually have some breakpoints thus would most likely end up in debug view to examine variables/call-hierarchy, etc; for testing, the test view did not provide any extra information that is not already accessible in the editor, therefore, I almost never need to switch to test view when running test via the gutter menu. In short, switching to debug view automatically is a useful/convenient feature since I will be going to debug view anyway; but switching to test view does not provide the equivalent usefulness (one can even argue it is destructive as it takes me away from where I wanted to be).

I understand your point about some users might not be aware that tests have been run, we actually have exactly the same discoverability issue. I ended up opening the test output terminal on startup instead, as it provides additional (more detailed) information than the test view or editor shown, and it helps the discoverability, at least for this extension. While the discoverability is a legit issue, I am just not sure showing test view on test run is the best solution...

Anyway, just throwing my opinion out there for whatever it's worth... BTW, I tried the latest insider build, with the change you mentioned above, I think it is much improved than the earlier insider. Thanks.

[CriusNyx]

This issue appears to have reappeared in the release on 6.2.0. I downgraded to version 5.2.3 and am not having any trouble.

Extension version: 6.2.0
VS Code version: 1.86.1 (Universal)
OS version: Sanoma 14.0 (23A344)
Machine: M1 Macbook Pro

[connectdotz]

Close this as a duplicate issue, see #1111.

Please see solution here