Sample Golden Scenario Status - Jupyter Notebooks

[burkeholland]

Scenario Status

Example walkthrough is in issue #2. However, feel free to come up with additional ideas or use cases. Just be sure to cover everything noted below.

Golden Scenario Legend

Status	Definition
🟢	Passing
🟡	Issue, but not blocking
🔴	Blocking

Overall Status Logic:

• If all items are passing, overall status is 🟢
• If items are passing or have non-blocking issues, overall status is 🟡
• If any items are blocked, overall status is 🔴

Issues:

• Create a log of issues below the table
• Use 💣 for blocking issues
• Use 🟡 for non-blocking issues
• Use 💎 for feature suggestions/improvements

Provide status for each subcategory (not required per issue)

Date	Overall Status	Local Setup	AzML Setup	Variable Explorer	Data Viewer	Running	Diffing	Language Server	Export	Live Share
09-05-2021	🔴	🟡		🟡	🔴	🔴	🔴	🟢	🟢	--
09-10-2021	🔴	🟡		🟡	🟢	🟡	🟢	🟢	🟢	🔴
09-17-2021	🔴	🟡		🟢	🟡	🟡	🟡	🟢	🟡	🔴
09-24-2021	🔴	🟢		🟢	🟢	🟡	🔴	🟢	🔴	--
10-03-2021	🟡	🟢		🟢	🟡	🟡	🟡	🟢	🟡	--
10-12-2021	🟢	🟢		🟢	🟢	🟡	🟢	🟢	🟢	--
10-17-2021	🟡	🟡		🟡	🟢	🟢	🟡	🟢	🟢	--
10-22-2021	🟡	🟢		🟢	🟢	🟡	🟡	🟢	🟢	--
10-31-2021	🟡	🟢		🟡	🟢	🟡	🟡	🟢	🟢	--
10-31-2021	🔴	🟢		🟡	🟢	🔴	🟡	🟢	🟢	--
11-15-2021	🟡	🟡		🟢	🟢	🟡	🟡	🟢	🟢	--
12-05-2021	🟡	🟢		🟢	🟢	🟢	🟡	🟡	🟢	--
01-12-2022 Part 1 Part 2	🟡	🟢		🟡	🟢	🟢	🟡	🟡	🟢	--
01-25-2022	🟡	🟢		🟢	🟢	🟢	🟡	🟡	🟡	--
01-31-2022	🟡	🟢		🟢	🟢	🟢	🟡	🟡	🟢	--
02-11-2022	🟡	🟢		🟡	🟢	🟡	🟢	🟡	🟢	--
02-20-2022	🟡	🟢		🟢	🟢	🟡	🟢	🟡	🟡	--
03-01-2022	🟡	🟢		🟢	🟢	🟢	🟡	🟢	🟢	--
03-02-2022	🟡		🟡	NT	NT	🟢	NT	🟢	NT	NT
03-01-2022	🟡	🟢		🟢	🟢	🟢	🟡	🟢	🟢	--
04-22-2022	🟡	🟢	NT	🟢	🟢	🟡	🟡	🟢	🟢	NT
05-13-2022 (Liveshare Special Run)	NA	NA	NA	NA	NA	NA	NA	NA	NA	🔴
06-06-2022	🟡	🟢		🟢	🟡	🟢	🟡	🟡	🟢	--

---

🔴 05-13-2022

@greazer - Did a special pass for testing Liveshare in notebooks and interactive window using Insiders. Overall LiveShare has problems that I was running into even outside of our features. These made it hard to know what might be specific to us.

Notebooks and scripts from https://github.dev/greazer/covid-19-data

• 🟢 Display notebook and typing in cell - Works both ways
• 🟡 Scroll notebook and type into a different cell. Host doesn't get any notification. Is this expected? Typing in guest does appear in the host though. Adding a new cell at nb end in host shows up in guest.
• 🔴 Bug found (vscode#149502): Couldn't page up or down in host past some cells. Not related to LiveShare.
• 🔴 Bug found(TBD): As host scrolls notebook or py script, guest should follow while in follow mode.
• 🔴 Bug found(TBD): Opened requirements.txt on host in existing LS session. No syncing. Tried opening requirements.txt on target, but file never loads
• 🔴 Bug found(TBD): Opening a notebook on guest and then forcing host to follow, kinda works, but scroll position is off as the guest is scrolled compared to what the host sees.
• 🔴 Bug found(TBD): Sometimes outputs sync from host to guest, sometimes not. Not sure when it works or doesn't.
• 🔴 Bug found(TBD): Kernel picker on guest says "Live Share Controller" (weird to call it Controller in this scenario).
• 🔴 Bug found(TBD): When host is set to working kernel, when I run a cell on the guest, nothing seems to be happening on host.
• 🔴 Bug found(TBD): After the above attempt to run a cell on guest, can't stop execution.
• 🔴 Bug found(TBD): Guest - No commands for showing variables view. No variables view at all, or DataViewer
• 🔴 Bug found(TBD): Guest - When guest has focus, no interactive window code-lenses but blue cell markers exist. Shift+enter seems to pretend to work, but doesn't. Create Interactive window is there, but doesn't do anything

Gave up. Too many problems.

🟡 03-02-2022

First focused run using Azure ML as the compute. While doing this was not blocked, there were several concerning issues encountered along the way. Issues being entered but here's indices into the video that reflect the trouble.

Try "Edit in VS Code" from Azure ML Studio

• 1:20 Bug 1 - Prompted to enable Azure ML extension which I already had installed, but wouldn't enable. Presumably due to the fact that a dependency (AzML remote extension) was disabled too. Unable to enable extension that depends on another disabled extension (AzML) vscode#144294
• 2:50 Bug 2 - After enabling, VS Code indicated trust issues that were resolved on reloading VS Code.
• 5:50 Bug 3 - Unknown Error retrieving Subscriptions (AzML Extension). Also prompted to reload VS Code. Just reloaded, but didn't reinvoke the URI. Known Bugs?

Try "Edit in VS Code" again.

• 7:00 Bug 4 - Even though we're signed into AzML in the Studio, we are prompted again to sign in after going to VS Code.
• 7:32 Bug 5 - After signing in again, get "Cannot read property 'subscription' of undefined. (AzML Extension). See: Cannot read subscription property vscode-tools-for-ai#1473.

Try "Edit in VS Code" again.

• 9:10 Everything seems to start as expected, finally
• 9:30 Because AzML extension activates, user is prompted to set a default workspace in the VS Code that was used to launch the remote VS code instance.
• 9:30-14:00 Bug 6 Discussion about whether there's something that can be done to avoid having two VS Code windows start for this scenario. Likely something to address in core codebase.
• 16:00 Confusion as to how the kernel is managed on transition from Studio to VS Code. New kernel? Connect to existing kernel?
• 21:00 Bug 7 Really really slow ipynb saving. Turns out to be an Azure ML Studio issue as well. Better feedback for VS Code being tracked here: No feedback when it takes a long time to save a good-size notebook in remote scenarios. vscode#144204
• 24:46 Bug 8 After closing first instance then doing scenario again, the new instance of VS Code that starts up to initiate the remote experience is actually a remote session. This is problematic if the remote compute was shut down.
• 25:56 Suggestion to solve above issue. ALWAYS open a new VS Code (local) window to initiate remote connection AND/OR don't open a new window if the launched VS Code is already attached to the right compute.

🟡 Overall Status for 2/20/2022

Blocking Issues

• 💣 Notebooks - Creating new notebook taking a long time vscode#134526
• 💣 Notebooks - Export to PDF never completes vscode-jupyter#7745
• 💣 Live Share doesn't work on Notebooks live-share#4427
• 💣 Interrupting an execution while in the run by line mode results in kernel timeout vscode-jupyter#8899
• 💣 https://github.com/microsoft/vscode-internalbacklog/issues/2753
• 💣 Interrupting a cell execution times out and am forced to restart kernel vscode-jupyter#9111

General

• 🟡 Native Notebooks - Moving non-consecutive cells only moves the second vscode#133915
• 🟡 Search doesn't work in text outputs vscode#141967

Setup

• 💎 Prompt or auto install requirements from requirements.txt vscode-python#4606
• 🟡 Notebooks - Kernel not maintained after save vscode-jupyter#8215

New File

• 💣 https://github.com/microsoft/vscode-internalbacklog/issues/2753
• 💣 Notebooks - Creating new notebook taking a long time vscode#134526
• 🟡 Two "New File" menus are confusing and notebook editor isn't loaded when language is set to "Jupyter"  vscode#135416
• 🟡 Notebooks - Create new notebook not honoring default kernel  vscode#134830
• 🟡 First cell should receive focus when a Notebook is created vscode-jupyter#9224

Variable Explorer

• 🟡 Variable explorer loading message is misleading vscode-jupyter#8898
• 🟡 Variable Explorer: Some variables are stuck in the loading state vscode-jupyter#7970

Data Viewer

• 💎 Visual clue for Data Viewer substring search vscode-jupyter#7628

Running

• 💣 Interrupting an execution while in the run by line mode results in kernel timeout vscode-jupyter#8899
• 💣 Interrupting a cell execution times out and am forced to restart kernel vscode-jupyter#9111
• 🟡 Rendered Markdown jumps / flickers as Notebook runs vscode#134388
• 🟡 Slight lag scrolling around large images in Notebook vscode#133005
• 🟡 Notebook keybindings don't work as expected when focus is in the webview vscode#133696
• 🟡 Notebooks: breakpoints unexpectedly moved when splitting cells vscode#137833
• 🟡 Notebooks: No visual indication that cells are pending on a cell in "Run by line" status vscode#137834
• 🟡 "Search" doesn't work inside interactive HTML reports vscode#143917

Diffing

• 🟡 Diff view shows output changes even when there is no actual change (only change is in the metadata) vscode#141966
• 🟡 Diffing view marks all cells after the newly-added cell as changed, decreasing its usefulness vscode#133645
• 🟡 Notebooks - Diff outputs staggered  vscode#135821
• 🟡 Notebooks - Reverting output change in diff does not revert input  vscode#134076
• 🟡 Diff view: output metadata change UX is confusing vscode#137836
• 🟡 Native Notebooks - Bit of screen cheese on diff view vscode#133914

Language Server

• 🟡 Error message persists for package imports pylance-release#2246
• 🟡 Error message appears in every cell vscode#140669
• 🟡 pylance working really slow. pylance-release#1338

Export

• 🟡 Exporting to HTML notification persists vscode-jupyter#9112
• 🟡 Toolbar ellipses replaced by Variable Explorer button vscode#138545
• 💣 Notebooks - Export to PDF never completes vscode-jupyter#7745 <- eventually exported successfully but took a really long time: 25'16''

Live Share

• 🟡 Really slow to connect to session with Notebooks live-share#4426
• 💣 Live Share doesn't work on Notebooks live-share#4427

---