Autocomplete hangs the kernel for interactive window

[stevenlis]

Version: 2020.1.58038 (13 January 2020)

I have a dataframe with shape of (810103, 26). If I type code and run a cell from the editor, everything is fine. But it will be extremely slow if I run code from the interactive window directly such as df.head(), df.columns. I can even make a screenshot without shortcut😂

[rchiodo]

That's really weird. We have reports of the opposite. Running from the editor being really slow, but directly from the interactive window being fast.

The problem is we can't repro in house. Do you have some code and data you can share?

[stevenlis]

I found the issue when cleaning this CDC dataset. Just read the csv file and then df.head()

I also tried with the toy dataset tips in seaborn, which didn't cause any problem. I'm using 2019.11.50794 now, which has no issues.

[rchiodo]

Hmm. I'm reproing. Only goes slow from the interactive window directly. And it's the kernel that's taking a super long time (not our code). But if I run it from the editor, df.head() is instant.

[rchiodo]

And we submit the same code in both cases. I'm guessing this is a kernel issue.

In the editor case, we run two executions actually:

*** REQUEST EXECUTE : {"code":"__file__ = 'd:\\\\Training\\\\SnakePython\\\\foo.py'","stop_on_error":false,"allow_stdin":true,"store_history":false} 
*** REQUEST EXECUTE : {"code":"df.head()\n","stop_on_error":false,"allow_stdin":true,"store_history":true}

whereas in the interactive window we only run the second. And running both of those in the interactive window has no effect on the execution, it still takes a long time.

[rchiodo]

And this is not a regression. It repros with 2019.11.50794

[rchiodo]

I take that back. Only sometimes with 2019.11.50794.

[stevenlis]

Hi, @rchiodo. just wondering... will this be fixed any time soon?

[rchiodo]

Not likely. We have no idea what causes it. It's jupyter that's taking all the time (not our code), so we'd have to debug the jupyter kernel.

[stevenlis]

@rchiodo I don't know what the advantage of this jupyter kernel thing but it adds some env I don't want:

I feel like I prefer the old extension approach better.

[rchiodo]

Those two new 'kernels' represent the environment you're running your python code in. They are an exact match for an environment generated by conda.

The advantage is that they will load all of the same packages that the associated conda environment loads.

There is no disadvantage other than they are listed there. They have nothing to do with the slowdown you're seeing.

The slowdown is in the IPython code itself (which is what I meant by we had to debug the jupyter kernel). The slowdown is not in the code we ship with the ms-python extension.

[stevenlis]

@rchiodo any process on this buddy? This happens so frequently and even with small datasets from time to time.

[DonJayamanne]

Do you have the variables list open when this issue occurs?

[stevenlis]

not at all I never used that thing

[rchiodo]

Sorry but we are not actively investigating this. I believe this is an issue in the IPython code itself and not our own. We don't do anything special to send code to the kernel.

[stevenlis]

Could you guys please take a look. This is so weird and really interrupt my workflow.

[rchiodo]

I have noticed if I interrupt the kernel when this is taking a long time it comes back immediately. At least for df.head()

[rchiodo]

Some of the advice here seems to alleviate the problem:
jupyter/notebook#3224

I typed this in to the interactive window:

%config Completer.use_jedi = False

and now it responds immediately. At least with the csv you mentioned above.

[rchiodo]

That config line causes Jupyter to stop giving us autocomplete information. So you won't get column names with autocomplete anymore, but that's probably worth it for the speedup

I take that back, it still seems to work. Not sure what the turning off of Jedi does then.

[stevenlis]

I don't even use Jedi but only Microsoft language server

[rchiodo]

This setting is not for us. It's for jupyter. They use jedi as well for autocomplete.

[stevenlis]

If I encounter this issue and try to Interrupt the kernel, the kernel will be forced to restart.
This is really annoying since sometimes it will take a long time to clean some data and you have to re-run everything.

[rchiodo]

So you're saying the config setting doesn't work?

%config Completer.use_jedi = False

[rchiodo]

And interrupting the kernel doesn't always reset the kernel. Depends upon the thing being interrupted.

[stevenlis]

So you're saying the config setting doesn't work?

I'm a little bit confused about it. Should I set use_jedi to False? Because based on your comments, it seems like you later found an issue with it. Is there any side effects of it?

And interrupting the kernel doesn't always reset the kernel. Depends upon the thing being interrupted.

This is why I don't expect to restart the kernel and now it always does.

[rchiodo]

Did you mean this comment?

That config line causes Jupyter to stop giving us autocomplete information. So you won't get column names with autocomplete anymore, but that's probably worth it for the speedup

I take that back, it still seems to work. Not sure what the turning off of Jedi does then.

Sorry that was confusing.

I was worried turning of jedi support in jupyter would prevent jupyter from returning autocompletions, but it does not.

So I don't see any side effects to this setting:

%config Completer.use_jedi = False

[stevenlis]

But this will turn off all the autocompletion while I enter code in the interactive window right? and do I have to do this whenever I launch the vscode or the kernel?

[rchiodo]

No. It doesn't. That's why I said this:

I was worried turning of jedi support in jupyter would prevent jupyter from returning autocompletions, but it does not.

After changing that setting I was still able to get column names from panda dataframes. This data is not provided by the python extension and explicitly comes from jupyter. So Jupyter still seems to provide autocomplete data even after its jedi usage is turned off.

[rchiodo]

You should be able to put this in your startup commands:

[rchiodo]

Finally had a chance to profile the IPython kernel. The autocomplete from typing is hanging the kernel and preventing the execution of a cell. We should be able to fix this by interrupting the kernel if it's in the middle of autocompletion.

[greazer]

Longstanding bug, and the workaround seems to alleviate the situation for the most part. Moving to October.

[DonJayamanne]

Why don't we keep track of the pending completion requests & let users know that things are slowing down because of Jedi in the kernel & ask them to just turn this off.

Some (if not most, totally guessing here) users will want this, to get better completion.
Feel its better to let them know when things are slowing down & what they can do.

Just like in VS Code core, they let users know what extension is chewing up CPU process & slowing VS Code down, then Users can optionally blitz that extension. Similarly users can opt into disabling jedi completely (we can just add that into a startup command if they click yes and if they want they can go back and just remove that line from the settings.json startup commands)

[rchiodo]

Yeah I wanted an automated solution. We didn't originally just make this a startup command because we never spent the time to investigate what the possible side effects of this were. Why is this not the default for jupyter in the first place?

[DonJayamanne]

ust make this a startup command because we never spent the time

I agree, we should get to the bottom of the issue, rather and applying a bandaid fix.

[DonJayamanne]

Proposed Solution

• Ensure we can repro this & find out the root cause
  • Find a good fix (why is it slow, can we change some settings in jedi)
  • Reach out to David Halter (we've worked with the author of Jedi closely in the past)
• Alternative
  • If intellisense response > n seconds, notify user about this so they can turn this off
  • If running cells is blocked because we're still waiting for completions, again notify user about the impact of this delay

[rchiodo]

It looks like we already implemented the 'disable jedi for kernels'. Code is here.

vscode-jupyter/src/client/datascience/jupyter/kernels/kernel.ts

Line 527 in e567b89

await this.disableJedi();

Closing this as I don't repro it anymore with that setting enabled.

[zrxdaly]

I also had same problem. After some searching online, I think i found how this problem occured and how to alleviate it.

This long hangs occured after I updated my conda version and at the same time my ipykernel also needs to update to higher version. I guess it is because of the version of ipykernel. In my case is: 6.4.1.

1. create a new virtual env
2. set ipykernel version as 5.3.4
3. before run code in cell, run conda install jupyter. This ensure vscode will not upgrade your ipykernel directly.