Nested IPython magics don't work

[LukaPitamic]

There is no great solution for working in isolated namespaces in Jupyter.
jupyter-spaces could solve this very elegantly.
Is there any workaround for magics to work?

[LukaPitamic]

Tried with programmatically running store magic...

Obviously get_ipython() always uses user name space instead of custom namespace

[davidesarra]

Hi @LukaPitamic! Thanks for opening this issue and sorry for my delayed reply. You’re right. At the moment Jupyter Spaces doesn’t support sub-nested magics. However, we can extend our support. IPython itself provides ways to support nested magic calls however we cannot always rely on them without breaking namespace isolation.

• Nesting line magics like %store under the %%space magic

  Example:

  %%time
  %%space xyz
  %store a
  alpha = 1.23

  We probably can add support for nested line magics because they execute within the wrapping cell magic so we can control the namespace.

• Nesting cell magics like %%time under the %%space magic

  Example:

  %%time
  %%space xyz
  %%time
  alpha = 1.23

  We probably won't be able to support nested cell magics because we need to defer the execution of the cell source to the inner cell magic thus losing control over the namespace.

  What we can do is raising an exception explaining the issue to the user and suggesting a workaround, that is placing the %%space magic after all other cell magics if possible.

Later this week or next week I’ll open a PR to tackle these two changes.

[LukaPitamic]

Hi @davidesarra, I totally agree with you on nested cell magics, but if it would be possible to make line magics work - this would provide incredible value. IMHO.

For example being able to use jupyter_spaces like:

%%space xyz
if 'alpha' not in locals() 
    alpha = 1.23
    %store alpha

opens so many new ways to use notebooks. Note the indent of %store.

And yes, absolutely - exception raised should provide valuable feedback about what is going on. Maybe for the time being even directing people to this threat.

Another idea might be to have an option to ignore all lines and/or cells which bump into this problem. try-except pass like. Which is by default False of course.

The way I'm trying to use jupyter_spaces is to run get_ipython() of cells in another notebook and add jupyter_spaces cell magic on top of the code in the corresponding cell. In this way I can run any notebook in its own name space without global namespace pollution (which often happens with our - data anlysis way of work). We often don't like to use custom functions which address this problem in coding world :D. And get_ipython() is always run in global namespace. Well - at least I wasn't able to persuade it other wise.

Please understand I'm talking from data analyst point of view. Magics like this one (and others of course) really provide such an elegant way to jump from idea to script fast. It is efficiency - thus I believe its worth to talk about how to move on with dev.

I deeply appreciate your contribution.

[davidesarra]

Thanks @LukaPitamic for describing your use case. It's very helpful :)

This and another issue recently raised led me to investigate alternative ways to segregate namespaces. I think we can fully rely on IPython methods to execute the cells while temporarily hijacking IPython's user namespace. In this way we can support both nested line and cell magics.

I also tried out the example you mentioned above on the new version I'm working on. I assumed that alpha in line 4 actually refers to a, i.e. a = 1.23. Let me know if it's not the case. Anyway, if it is, that example appears to work.

[LukaPitamic]

Yes, you are correct - I corrected the code (a and alpha) in example above.

Wow, I can’t wait for new version.

[LukaPitamic]

@davidesarra is there any way for me to play with your

new version you're working on

I'm just at the point of starting to implement jupyter_spaces into my workflow. Maybe you could use some beta-testers. Especially if %store magic works as expected like you mentioned.

[davidesarra]

@LukaPitamic Over the last week I tried to use IPython's interactive shell directly to trigger the execution by swapping namespaces. However, that brings up a regression bug (a function defined in a cell cannot use variables from its enclosing scope).

So I went back to evaluate the independent execution route, but that turned out to be non-viable for your use case because the execution of nested both cell and line magics should be handed over to IPython's interactive shell.

Sorry for the bummer. I'm continuing to explore more IPython's codebase to understand how to avoid that regression bug and be able to deliver this feature. Hopefully there is a viable way.

I'll update this issue when I know more. I'd be very happy to have somebody try out the commit in PRs. Thanks for your patience :)