Benchmarks

[rebornix]

To get a better understanding the overall performance of opening/loading a notebook with different implementations, we will do some benchmark with different types of Notebook files, the matrix is

• Platform
  • Python Notebook
  • Azure Data Studio
• Notebook
  • Regular size sample
    • 10 code cells and 20 markdown cells
  • Medium size sample with duplications
    • 80 code cells and 200 markdown cells
  • Large file
    • 545 code cells and 1292 markdown cells

There are a couple of factors that affect the overall performance. To make sure we get the right numbers, we need to track the benchmarks in following aspects

• Overall loading time, from user clicking the file to a fully usable state
• Initialization time, from user clicking to first paint.
• View/Editor loading time, from first pain to a fully usable state.

Results

All tests are run multiple times to get fair results. Benchmark profiles are measured by Chrome Dev Tools Performance tool. Before each benchmark, garbage is always collected. For more details of each benchmark's water flow, please read the comments below.

Platform	File Size	Initialization	Cells Rendering	Total
Python	Regular	3.88s	1.84s	5.75s
	Medium	4s	6.28s	11s
	Large	4.1s	74s	80s
ADS	Regular	250ms	443ms	0.8s
	Medium	210ms	5s	6.43s
	Large	600ms	108s	120s

Interesting findings:

• Python notebooks always take 3-4 seconds to initialize the webview while ADS opens file almost instantly
• Once the initiation finishes, the rendering time is similar. This is as expected because most of the time is spent in rendering Monaco Editor and markdown content
• For the large file, Python's impl opens the file more stably while ADS often freezes and takes longer to open the file. Once opened, I can scroll with noticeable latency in both but the Python one is more usable.
  • The reason behind this might be that Python's notebooks is running in a webview which runs in its own process.
• No virtualization in both solutions.
• Memory usage are similar.

[rebornix]

--Python--

Regular

• From clicking file name to web view loaded successfully and connection established: 3s
• Render the notebook: 1s (rendering 9 Monaco instances)
  • Monaco Editor takes 600ms
    • 400ms in scripting
    • 200ms in layout

Medium

• From clicking file name to web view loaded successfully and connection established: 4s
• Render the notebook: 7s
  • 5s in scripting
  • 2s in rendering

Large

• Render the notebook: 90s. Most of the time is used in layout (the red ranges are all forced reflow layout)
  • 20s is spent in scripting.
  • Scrolling is slow but it's fairly usable.

[rebornix]

--Azure Data Studio--

Running Code sample

• From clicking file name to web view loaded successfully and connection established: 40ms
• Render the notebook: 600ms (rendering 9 Monaco instances)

Medium

• 5s in total
  • No noticeable latency between clicking and file opening
  • 2.7s for scripting, and 2.2s for rendering/layout

Large

• No noticeable latency between clicking and file opening
• Window freezes two out of three times of testing. It usually take 100~120s if loaded successfully
• Scrolling is painfully slow (much slower than Python's webview impl) and it's not usable.

[rebornix]

Where the time is spent

The scripting/layout time ratio is around 20/80. It indicates that the scripts trigger forced reflow layout very often. If we dig into the water flow of ADS benchmarks we can find some proves

Firstly, editor and markdown content creation. This phase takes 70s for opening the large notebook.

In above frame, a monaco editor control is created and as it doesn't provide the initial dimension, it attempts to measure the container DOM node size, which takes 30ms. The layout time accounts for 3/4 of the editor creation time.

And then a renderMarkdown call triggers another forced reflow, which takes another 30ms. The scripting time renderMarkdown is only 2ms.

For the large file, we have 545 code blocks and 1292 text cells, the time spent in forced reflow is at least 545*30 + 1292*30 = 55,000 ms.

Secondly, the second phase is all editor updating and relayout.

The resize observer we put on every monaco editor container detects some size change and triggers a recompilation of the editor options, which lead to a relayout finally. In the water flow, what we see is a 2ms editor option update followed by 50ms browser forced reflow.

With 545 editors, it's about 545*50 = 27,250ms.

[rebornix]

The performance of opening a notebook file can be improved by optimizing editor creation and layout, especially for the initial creation. To get a better understanding of how this can be improved, here are some attempts I've done

Layout editor with calculated dimensions

The size of a monaco editor is determined by line count of the text model. Since we know the height of a view line, we can get the height of the editor by calculation. As we pass in the calculated dimensions when running editor.layout({ width, height }), the file opening time is reduced to 30s.

It's better than before, but still not acceptable as open the same file in text editor takes hundred of ms. Scrolling is terrible as well since there are too many DOM nodes.

Virtualization

After adopting List View, the file loading performance is improved significantly as the List View is virtualized. It loads several view ports into the DOM and remove unused ones when necessary.

For loading a file with 200 code blocks and 1000 markdown cells, it takes 350ms ... actually it only loads the first 20 of them.

The scrolling is much better than loading whole content into view, but it's still not good enough. Each onScroll handling takes around 200 - 600ms depends on how many code cells are loaded in future viewports. For comparison, you can open Interactive Playground and scroll in it, you won't feel any lag at all.

Some ideas of improving scrolling

• Editor
  • Use simple editor to limit editor contributions
  • 🏃 Cache options/contribs/etc loading
  • 🏃 Lazy load more editor contribs (Suggest Controller is costly for example)
• List View
  • Add timeout for onScroll handler
  • Remove nodes from the tree less eagerly. List View currently keeps 20 nodes in the notebook scenario. Removing nodes is fast but removing nodes and then adding them back is costly. If users are scrolling up and down, we don't want to repeat removing and adding nodes which are never visible to users.
  • Adding future viewports into the DOM node when idle.
  • Rendering nodes in separate layers. Not sure how much we can gain from this but the idea is reducing layout/rendering time.

Simple editor, caching, translate3d

The scrolling is below 200ms on average. The lag is still noticeable sometimes.

List item dynamic height

We use dynamic height option in the List View at the beginning so it will calculate the list item height every time when a new list item needs to be added to the tree. However, this step is costly as it will add the node to a temporary tree and then remove it from the tree. In below water flow, you can see that monaco editor is created to calculate its height and then be removed.

	private probeDynamicHeight(index: number): number {
		const item = this.items[index];
                ...
		row.domNode!.style.height = '';
		this.rowsContainer.appendChild(row.domNode!);

                ...
                this.rowsContainer.removeChild(row.domNode!);
       }

However, the height of each editor is predictable: lineHeight * lineNumber so we can use dynamic height for markdown cells only. After the optimization, we reduce the list rendering near half.