Support large documents in extension host

[emackey]

• VSCode Version: 1.14.1
• OS Version: Windows 7 and Windows 10, likely all OSes

When editing documents with very long lines, such as lines containing data-URIs, the property vscode.window.activeTextEditor will be undefined when it should be defined.

Steps to reproduce:

1. Create a new (blank) VSCode extension, and add this command to extension.ts:

'use strict';
import * as vscode from 'vscode';

export function activate(context: vscode.ExtensionContext) {
    let disposable = vscode.commands.registerCommand('bigDocTest.runDocTest', () => {

        let hasActiveTextEditor = (vscode.window.activeTextEditor !== undefined);

        if (hasActiveTextEditor) {
            vscode.window.showInformationMessage('Success, activeTextEditor is defined.');
        } else {
            vscode.window.showErrorMessage('Oops, activeTextEditor is not defined.');
        }
    });

    context.subscriptions.push(disposable);
}

export function deactivate() {
}

2. Launch this extension, load any normal text file, and run this command, you see the "Success, activeTextEditor is defined" information message appear.

3. But, try loading a file with a very long line contained inside it, such as this one: MetalRoughSpheres.gltf. This file has long lines due to the use of embedded data URIs in the file. If you run the above sample extension while editing this file, you see "Oops, activeTextEditor is not defined."

This makes it difficult for VSCode extensions to work with such files. I'm the lead dev of the GLTF extension for VSCode, and this prevents my extension from running on GLTF files that embed large images as data URIs.

[jrieken]

Yes, we unfortunately have to do that because very large documents (those with long lines or many lines) are very likely to make the extension host choke and stall. The current limit is at 5MB. If a file exceeds that we don't sync it over to the extension host but only offer editing capabilities.

[emackey]

Thanks for replying. Does the user have any way to configure the 5MB limit to be larger? Could this issue be turned into an enhancement request for user-configurable max line length for extensions?

The current limit is particularly painful for extensions whose sole purpose is to augment editing of such files.

[jrieken]

The limit isn't configurable and it is somewhat arbitrary, tho before increasing it we should check how the extension host behaves with large files. Obviously this is more complicated then just having a check for one file, because many little files can also be a problem...

[emackey]

Great, thanks @jrieken. GitHub isn't showing me a reopen button, can this be reopened?

[schaveyt]

I am working on a extension that must work on large xml files which routinely are over 5MB. I just ran into this issue :-(. Making this configurable by the extension or offering an alternative scheme for commands to to still work would be helpful.

[jrieken]

re #31078 (comment) - forgot to press reopen...

[emackey]

Anyone happen to know where this limit is specified in the source code?

[schaveyt]

Perhaps a very limited, restricted api. IMHO, the primary thing that I and other should be able to get by with is:

Request the Uri and LineNumber of the Cursor for the active document.

The intent is for the vscode to tell the extension author, "Hey here is the what the user is up to, but I am not parsing that big file for you..have fun :-)"

window.activeTextEditorLocation(): Location | undefinded

Example usage:

'use strict';
import * as vscode from 'vscode';
import {MyCustomExtensionHelper} from './MyCustomExtensionHelper'

export function activate(context: vscode.ExtensionContext) {

    let disposableWordCounts = vscode.commands.registerCommand('bigDocTest.runDocTest', () => {
        let activeTextEditorLocation = (vscode.window.activeTextEditorLocation !== undefined);

        if (!activeTextEditorLocation) {
            vscode.window.showErrorMessage('No document is open.');
        }

        const helper = new MyCustomerExtensionHelper();
        helper.reportWordCount(activeTextEditorLocation.uri)   
    });

    let textEditorCommand = vscode.commands.registerTextEditorCommand('bigDocTest.customMenuItem1', () => {
        let activeTextEditorLocation = (vscode.window.activeTextEditorLocation !== undefined);

        if (!activeTextEditorLocation) {
            vscode.window.showErrorMessage('No document is open.');
        }

        const helper = new MyCustomerExtensionHelper();
        const activeWord = helper.getWordTheUserCursorIsOne(activeTextEditorLocation);
        helper.findAllReferences(activeWord);
    });

    context.subscriptions.push(disposableWordCounts, textEditorCommand);
}

export function deactivate() {
}

This one little function would enable a fairly wide variety of use cases.

• This would fix my ReferenceProviders
• This would fix my DefinitionProviders
• This would fix my TextEditorCommand handlers
• This would fix my Command handlers

[vscodebot]

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[Leonid-Kokhnovych-Rivian]

It would be great for the user to be able to configure it at least. I work with log files of max 200MB using Visual Studio Code and since there is no native filtering using line or regex (include/exclude), I can't filter the log file through third-party extension either...

[mbehr1]

@Leonid-Kokhnovych-Rivian please see my extension https://github.com/mbehr1/vsc-lfs. It's really hackish but might be useful for your use-case especially if you want to filter a lot of content anyhow (see setting vsc-lfs.replacements ).

[zb3]

@emackey, this is controlled by the _MODEL_SYNC_LIMIT property of the TextModel class defined here:

vscode/src/vs/editor/common/model/textModel.ts

Line 176 in ecf479e

static _MODEL_SYNC_LIMIT = 50 * 1024 * 1024; // 50 MB, // used in tests

On my system I was able to increase this limit by patching the workbench.desktop.main.js file. However, note that this might cause vscode to display a warning that the installation is corrupt. Still, this worked for me.

Here's the command I used (not that I recommend doing it):

$ sudo sed -iE 's/\._MODEL_SYNC_LIMIT=50\*1024\*1024,/._MODEL_SYNC_LIMIT=500*1024*1024,/g' /usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js

[ThaDaVos]

Any progress on this? I want to use "Text Power Tools" (https://github.com/Microsoft/vscode/issues/31078) to filter large LOG files (size of them is out of my control) - and not having this function makes me to use Powershell to filter them before hand instead - which is a petty - as I rather use it inside vscode - when can we expect a fix for this?

[RossBrunton]

Just stumbled into this issue myself. I have an extension that gives me emacs style keybindings. Of course, since it's an extension, whenever I open a file larger than 50MiB, they don't work. This makes me unable to navigate large files with the keyboard (even through the cursor keys).

An option to configure _MODEL_SYNC_LIMIT would go a long way. I'd see about implementing it myself, if Microsoft would be willing to merge it.

[oaldrian]

It's been 5 years now with this issue. Can we please get the possibility to edit the limit in the config. The default can stay the same, so most people wouldn't even notice.

[ada2k]

It would be very useful to at least have find and replace for very large lines. Currently i have to use grep

[yachnyymaxim]

The issue is still in place, and it would really be nice to make that limit configurable and let users decide whether they are ok with excessive CPU/MEM usage.